Freeze a Portable Evaluation Harness

Week 4 left R4-trace-schema, 45 cases, and a causal failure ledger. Manual review found useful failures, but changing fixtures or thresholds between runs can make two agent versions incomparable. This week you freeze a portable evaluation harness: a dataset supplies cases, a solver runs the candidate, and multiple scorers evaluate independent claims.

Portable means the case schema and core scorers run without one hosted evaluation product. Use a framework-neutral CLI as authority and an Inspect AI adapter for orchestration. OpenAI’s hosted Agent Builder and Evals are scheduled to stop being available after November 30, 2026, according to its June 3, 2026 update; durable concepts must not depend on that hosted surface.

At the outer boundary, the harness is a pure experiment coordinator: frozen cases enter, an isolated solver runs, independent results leave. The candidate cannot inspect expected answers or mutate thresholds.

The candidate runtime at agent.py is only the solver target. It cannot read reference answers, expected effects, or grader thresholds.

Every case pins tenant, actors, policy/source versions, initial state, request, available tools, approval state, expected outcome, prohibited effects, decisive evidence, class, risk, and reference solution. A suite digest must cover ordered case content and schema version; editing either must create a new suite. Tagged v1.0.2 hashes parsed cases but not an explicit schema-version field, so schema-version binding remains a design target rather than proved behavior.

state.py resets named disposable state before each case. Isolation between trials prevents an earlier successful mutation from making a later case pass accidentally.

The frozen loader in evals/dataset.py rejects empty cases, missing decisive assertions, duplicate IDs, mutable thresholds, and references that fail their own case.

def load_cases(path: Path = CASES_PATH) -> tuple[EvalCase, ...]:
    cases = tuple(
        EvalCase.from_dict(json.loads(line)) for line in path.read_text().splitlines() if line
    )
    if not cases:
        raise ValueError("dataset must contain at least one case")
    if len({case.case_id for case in cases}) != len(cases):
        raise ValueError("dataset case IDs must be unique")
    return cases

Canonical serialization makes suite identity content-addressed. Tagged evals/dataset.py sorts object keys and removes presentation whitespace before hashing:

def dataset_digest(cases: Iterable[EvalCase]) -> str:
    canonical = json.dumps(
        [asdict(case) for case in cases], sort_keys=True, separators=(",", ":")
    ).encode()
    return sha256(canonical).hexdigest()

json.dumps is the canonicalizer, SHA-256 is the digest function, and disk stores the frozen JSONL plus resulting digest. Editing case order or content changes the hash; changing only Markdown prose does not. This proves input identity, not dataset quality, so reference solutions and negative controls remain separate gates.

Close the tagged schema gap by hashing an explicit envelope in your exercise implementation:

payload = {
    "schema_version": "eval-case-v1",
    "cases": [asdict(case) for case in cases],
}
suite_digest = sha256(
    json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()

This snippet is the course’s design target, not byte-for-byte tagged source. Pinning parser/package versions still matters because canonical serialization behavior is part of the digest contract.

Grow to 75 cases across normal, boundary, failure, and adversarial classes. Do not chase equal counts yet; preserve risk coverage and record denominators.

A harness coordinates evidence but must not collapse it. Business outcome, hard safety, task quality, trajectory, RAG, reliability, operations, and diagnosis remain separate outputs.

Before running the 75-case enterprise harness, use the supplied six-case support/refund project as a compact executable starter. It demonstrates the same dataset → agent → structured trace → independent scorer shape without an API key or third-party dependency. Download the exact ZIP; its source discussion is Teach AI Evaluation Example.

PROJECT_ROOT="$PWD"
STARTER_ZIP="$PROJECT_ROOT/ai-agent-evals.zip"
LAB_DIR="$PROJECT_ROOT/.tmp-ai-agent-evals-lab"
STARTER_RECEIPT="$PROJECT_ROOT/ai-agent-evals-starter-results.json"
RECEIPT_TMP="$PROJECT_ROOT/.ai-agent-evals-starter-results.json.tmp"
EXPECTED_SHA256='9266eca1d9b6b187a0b6295611bd1f0a6c282281bcd253c36366dcb898f52388'
LAB_STATUS=0

(
  set -euo pipefail
  test -f "$STARTER_ZIP"
  test ! -e "$STARTER_RECEIPT"
  test ! -e "$RECEIPT_TMP"

  ACTUAL_SHA256="$(shasum -a 256 "$STARTER_ZIP" | awk '{print $1}')"
  if [ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]; then
    printf 'SHA-256 mismatch; refusing to extract\n' >&2
    exit 1
  fi

  test ! -e "$LAB_DIR" || {
    printf 'Disposable lab directory already exists; refusing to overwrite it\n' >&2
    exit 1
  }
  mkdir -- "$LAB_DIR"
  cleanup() {
    test "$LAB_DIR" = "$PROJECT_ROOT/.tmp-ai-agent-evals-lab"
    test "$RECEIPT_TMP" = "$PROJECT_ROOT/.ai-agent-evals-starter-results.json.tmp"
    rm -rf -- "$LAB_DIR"
    rm -f -- "$RECEIPT_TMP"
    test ! -e "$LAB_DIR"
  }
  trap cleanup EXIT

  unzip -q "$STARTER_ZIP" -d "$LAB_DIR"
  cd "$LAB_DIR/ai-agent-evals"
  python3 src/eval_demo.py --out "$RECEIPT_TMP"
  PYTHONPATH=src python3 -m unittest discover -s tests -v
  # Expected: 3 tests run, all OK.
  mv -- "$RECEIPT_TMP" "$STARTER_RECEIPT"
) || LAB_STATUS=$?

test "$PWD" = "$PROJECT_ROOT"
test ! -e "$LAB_DIR"
if [ "$LAB_STATUS" -ne 0 ]; then
  test ! -e "$STARTER_RECEIPT"
  printf 'Starter evaluation failed; no receipt accepted\n' >&2
  false
else
  test -f "$STARTER_RECEIPT"
  printf 'Starter receipt: %s\n' "$STARTER_RECEIPT"
fi

Parentheses create a subprocess, so strict shell options, exit, the trap, and cd cannot alter your interactive shell. The digest mismatch exits that subprocess before the disposable directory exists or unzip runs. After both demo and tests pass, the subprocess atomically moves the result to $PROJECT_ROOT/ai-agent-evals-starter-results.json; the caller then proves its working directory is unchanged, the lab is absent, and the positive-control receipt exists. The exit trap removes only the exact lab and temporary receipt paths after success or later failure; existing output is never overwritten. The ZIP README uses python; use python3 where that is the installed command. Do not edit the archive to normalize this environment difference. On the exact six cases, v1 records task success 0.6667, financial safety 0.8333, Arabic-billing task success 0.0, and a failed gate. Version 2 records 1.0 for every reported overall and slice metric and passes the demo gate. These results prove the starter mechanics on frozen local inputs; they do not prove production refund effects, tenant isolation, representative coverage, or CI execution.

The starter Trace keeps final status, answer, tool, transaction ID, amount, accessed tenant, authentication-before-tool, and turn count separate. Inspect those fields before reading aggregate scores: a fluent refund message cannot substitute for the correct transaction, amount, tenant, or authentication order. Then keep the six-case report as a small positive control while moving into the richer enterprise schema below.

The tagged framework-neutral runner in evals/task.py preserves independent result lanes. An optional Inspect adapter should mirror this authority rather than replace it:

        scores = {
            "outcome": asdict(outcome_score),
            "safety": asdict(safety_score),
            "trajectory": asdict(score_trajectory(case, result)),
            "rag": asdict(score_rag(case, result)),
            "judge": asdict(score_communication(case, result)),
        }

Open the harness black box and four ownership boundaries appear. Dataset loader owns immutable inputs, state factory owns isolation, solver owns candidate behavior, and scorers own verdicts; result writer only persists what those components produced.

Run authority stays one command:

python -m evals.task --output reports/suite-results.json
python -c 'from hashlib import sha256; from pathlib import Path; print(sha256(Path("reports/suite-results.json").read_bytes()).hexdigest())'

The first command allocates fresh in-memory state per case, spends local CPU, and writes a JSON report to disk. The second hashes the full artifact; compare the stable suite_digest inside reports across runs, while allowing created_at to make whole-file hashes differ.

Run twice against deterministic adapters. Frozen inputs and deterministic solver should produce identical result digests. A model-backed solver may vary, but case, policy, tool, agent, grader, and environment versions must remain pinned so variance is interpretable.

Execute a normal restore and an unauthorized mutation denial with an unaffected Globex positive control. Then introduce a case whose expected final state already matches initial state. A do-nothing solver may pass, revealing a vacuous evaluation. Recovery changes initial state or adds decisive trajectory/effect evidence, proves the reference passes, and proves the no-op negative control fails.

Broken decisive instrumentation returns NO_VERDICT; it never becomes a safety pass. Cleanup resets each disposable state store and waits for every child/process exit while retaining immutable result digests.

The scored checkpoint gives one ambiguous case; the unknown is the minimal repair. Expected evidence is one-dimensional: reference passes and named negative control fails. Feedback shows that a 0% or 100% result can reveal a broken task or evaluator rather than agent quality. Decline proposals that add an agent feature while repairing the case.

Spend 1.5 hours on Anthropic’s agent eval guide and the Inspect tutorial, 4–5 hours on schema/harness/cases, and 1 hour documenting reproducibility. The handoff is R5-suite-v1, D75-cases, suite digest, and two harness logs. Week 6 adds independent graders and calibrates the only subjective judge.