Trace the First Causal Failure

Week 3 produced R3-tenant-proof, 40 cases, and a tenant-scoped policy corpus. A failed final answer still does not say whether the first error was misunderstood intent, bad retrieval, wrong tool arguments, rejected execution, or incorrect state interpretation. This week you add correlated traces—ordered records of operations inside one run—and label the earliest event that made success impossible.

A span records one operation with start/end time, status, bounded attributes, and parent relationship. Trace the model decision, retrieval, tool validation and call, approval check, state read/write, final verification, latency, tokens, and estimated cost. Do not record raw prompts, secrets, policy bodies, or personal data.

The smallest diagnostic system has three boxes: runtime emits ordered evidence, trace plane preserves causal order, and diagnosis selects the first failed boundary. Final output remains evidence, but it is not automatically the cause.

Tagged agent.py does not yet wire the trace recorder through each decision. Adding one run ID and preserving parent/child order is today’s design target so a late communication failure remains distinguishable from an early wrong employee choice.

One giant “agent.run” span hides the boundary that failed. Instrument work where authority and state change: retrieval owns candidate evidence; broker owns validation and tool calls; adapter owns authoritative effects.

The tagged retrieval.py proves authorization-before-ranking; retrieval-span enrichment remains a design target:

    scoped = (policy for policy in policies if policy.tenant_id == tenant_id)
    terms = {term.casefold() for term in query.split()}

tools.py supplies typed authorization errors and effect receipts. Adding tool name, schema version, argument digest, authorization result, effect ID, and latency to allowlisted spans is today’s design target. An argument digest proves which validated input ran without leaking its contents.

The recorder at telemetry.py stores redacted, OpenTelemetry-shaped spans in process. It does not export them; wiring a real OpenTelemetry SDK/exporter while preserving the allowlist is a design target. OpenTelemetry’s semantic conventions define common telemetry names; GenAI conventions are evolving, so pin the package/convention version and keep application-specific attributes bounded.

    def record(self, kind: str, status: str, attributes: Mapping[str, object]) -> TraceSpan:
        safe = {key: value for key, value in attributes.items() if key in ALLOWED_ATTRIBUTES}
        span = TraceSpan(self.run_id, len(self.spans) + 1, kind, status, safe, monotonic_ns())
        self.spans.append(span)
        return span

Open the trace plane and causality becomes an ordered comparison, not a guess. Each operation records a monotonic sequence; classifier walks that sequence once and stops at first error.

Multiple symptoms can follow one mistake. If retrieval selects a stale policy, the model proposes the wrong permission and the final message becomes false; retrieval is the earliest causal failure. Record exactly one primary label plus supporting span IDs:

1. intent misunderstanding; 2. planning; 3. retrieval; 4. tool selection; 5. tool arguments; 6. execution; 7. state interpretation; 8. recovery; 9. authorization or safety; 10. final communication; 11. evaluation defect.

An evaluation defect means the system succeeded but the case, reference, instrumentation, or grader was wrong. It prevents teams from “fixing” a healthy agent to satisfy a broken test.

Tagged failures.py implements the selection rule directly:

def classify_earliest(spans: tuple[TraceSpan, ...]) -> FailureCause | None:
    for span in sorted(spans, key=lambda item: item.sequence):
        if span.status == "error":
            return _KIND_TO_CAUSE.get(span.kind, FailureCause.EXECUTION)
    return None

Python sorts in memory by sequence, then scans until first error. Complexity is O(n log n) because of sorting; if recorder already guarantees ordered immutable spans, a validator plus linear scan can reduce this to O(n). Evidence is one enum label tied to one span sequence, not a list of downstream symptoms.

The case loader in evals/dataset.py supplies frozen inputs. Produce a separate 20-run diagnosis table with run ID, outcome, earliest label, decisive span, component, and reviewer note; that table is an exercise artifact, not present in the tag. Add broken tracing, missing parent, truncated tool result, stale state interpretation, and evaluator-defect controls during the staged 45-case milestone.

Run a normal access restore and a missing-approval denial, with Globex unchanged as positive control. Then drop the retrieval span. The result may be correct, but causal diagnosis must return NO_VERDICT where decisive evidence is absent—not guess from the final message.

Recovery restores instrumentation, replays the same frozen case under a new run ID, and checks parent order plus state outcome. Cleanup deletes only disposable trace exports; case and run digests remain. Keep a local analogue caveat: one process can prove correlation logic but not distributed collector delivery, retention, or production privacy.

Run the focused tagged proofs:

python -m pytest tests/test_telemetry.py tests/test_failures.py -q

The scored checkpoint supplies one complete trace; the only unknown is earliest causal label. Feedback contrasts the worst visible symptom with the first wrong boundary. Decline multiple primary causes. Spend 1.5 hours with the Phoenix tracing tutorial and Hugging Face observability unit, 4–5 hours instrumenting and analyzing, and 1 hour writing the failure ledger.

The handoff is R4-trace-schema, D45-cases, and a 20-run failure ledger. Week 5 consumes these receipts to build a portable dataset → solver → scorer harness.