Distinguish Capability from Reliability

Week 7 produced 85 frozen cases and separated retrieval from generation. A stochastic agent can pass once and fail the next four identical trials, so a single run measures neither repeatability nor operational cost. This week you run every case five times under pinned versions and report capability and reliability as different questions.

pass@1 is the probability one sampled trial passes. pass@5 asks whether at least one of five passes—useful for “sometimes capable.” pass^5 asks whether all five pass—useful for “consistently reliable.” State assumptions and sample limits; five trials reveal instability but do not establish a precise production rate.

Keep repeated execution as one black box first. Frozen case and version digests enter; five independent trial receipts leave. Aggregation happens afterward so no failed trial disappears inside an average.

agent.py runs each frozen input under the same agent, model, tool, policy, and budget versions. Each trial starts from independently reset state.

Summary statistics erase which case and trial failed, blocking diagnosis and replay. Preserve each reset, version set, outcome, latency, token count, and cost before computing any aggregate.

evals/dataset.py stores all 425 trial records, not only averages. Slice by task type and risk so easy normal cases cannot hide flaky approval or tenant-boundary cases.

telemetry.py defines allowlisted latency, token, and cost attributes. The deterministic v1.0.2 report records zero placeholders for them; provider-backed measurement remains a design target. These are operability dimensions, never substitutes for outcome or safety.

Open repeated execution and independence becomes concrete. State resets before every trial; raw booleans and operational measurements are stored before statistics are computed.

One pass rate cannot answer both “can this agent ever succeed?” and “will it succeed every time?” Compute capability, consistency, and variance from the same raw trials without treating them as interchangeable.

evals/task.py groups outcomes without inventing one overall score:

Tagged evals/reliability.py computes the three views from the same raw boolean vector:

def reliability_metrics(passes: list[bool]) -> dict[str, float]:
    if not passes:
        raise ValueError("at least one trial required")
    values = [float(value) for value in passes]
    return {
        "pass@1": sum(values) / len(values),
        f"pass@{len(values)}": float(any(passes)),
        f"pass^{len(values)}": float(all(passes)),
        "variance": pvariance(values),
    }

any answers whether capability appeared; all answers whether consistency held. pvariance measures instability in this observed sample. Python runs these reductions in memory; stored per-case vectors remain authority for later slicing and audit.

Thresholds chosen after seeing candidate results turn release policy into storytelling. Freeze trial count, safety invariants, cost cap, and later provider latency floors before comparison begins.

Tagged release/policy.yaml predeclares five trials, zero critical failures, digest equality, and a zero-cost offline budget. Provider latency and risk-specific reliability floors remain design targets. A safety failure remains an immediate reject even if all averages improve.

The tagged release declaration keeps sample count and hard gates machine-readable:

suite:
  dataset: frozen-v1
  required_cases: 100
  required_trials: 5
hard_gates:
  critical_safety_failures: 0
  broken_decisive_evidence: NO_VERDICT
operations:
  max_cost_usd: 0.00

The release comparator interprets this policy; YAML itself changes no runtime state. It becomes effective only when CI loads it and rejects nonconforming evidence. CPU evaluates gates, disk holds policy/results, and provider runs would additionally consume network plus model capacity.

Generate the tagged five-trial artifact and exercise formula properties:

python -m pytest tests/evals/test_reliability.py -q
python -m evals.reliability --trials 5 --output reports/reliability-results.json

The runner creates 425 isolated case-trial executions for 85 cases at this week’s milestone. Inspect by_case before summary fields; a retained raw vector is the proof that one successful sample was not mistaken for consistency.

Run a stable normal case and a five-run missing-approval denial, with Globex as unaffected control. Then inject a flaky timeout on one trial. Recovery fixes or bounds the timeout, reruns all five under a new agent version, and preserves both result sets. Cleanup stops all child processes and resets trial stores while retaining immutable raw-result digest.

The scored checkpoint supplies a five-outcome pattern; the unknown is the statistic that answers “sometimes capable” or “consistently reliable.” Feedback contrasts one success in five with five successes in five. Decline claims of statistical significance without intervals or sample caveats. Spend 1.5 hours on τ-bench and GAIA, 4–5 hours on property tests plus 425 trials, and 1 hour on risk-sliced interpretation.

The handoff is R8-reliability-baseline and a release budget. Week 9 turns critical security properties into hard, non-averagable release gates.