Measure Retrieval Before Generation

Week 6 left calibrated independent graders and a 75-case suite. A wrong policy answer can still originate in two different systems: retrieval may omit the decisive passage, or generation may ignore a passage that was present. This week you record retrieved IDs separately from generated claims and effects, then reach 85 cases with irrelevant, stale, conflicting, injected, and foreign-tenant documents.

Recall@k asks whether the required passage appears among the top k allowed results. Context precision asks how much retrieved context was relevant. Groundedness checks whether claims follow supplied evidence; citation correctness checks whether each citation supports its claim. Tenant-leak rate is a hard zero-tolerance invariant.

Begin with two independently observable black boxes. Retrieval receives authorized candidates and emits IDs; generation receives those passages and emits claims, citations, and proposed effects. A bad final answer does not identify which box failed.

The policy authority in contracts.py pins tenant and as-of version. Stale/conflicting same-tenant documents are test inputs; foreign documents remain unauthorized.

Create controlled corpus variants: one irrelevant near-duplicate, one stale superseded rule, two conflicting current passages, one indirect prompt injection, and one colliding Globex policy. Changing one variable at a time preserves causal diagnosis.

Tagged retrieval.py returns tenant-scoped passages with citation IDs. Persisting scores and corpus digest beside those IDs is this week’s extension:

def scoped_policy_search(
    policies: tuple[PolicyPassage, ...], tenant_id: str, query: str, limit: int = 5
) -> tuple[PolicyPassage, ...]:
    """Authorize namespace before ranking; retain stale/conflicting versions for diagnosis."""
    scoped = (policy for policy in policies if policy.tenant_id == tenant_id)

If required evidence is absent, classify retrieval. If it is present and the answer contradicts it, classify generation or state interpretation. Never tune retriever and prompt in the same experiment.

Opening both boxes reveals two different comparisons. Retrieval metrics compare expected IDs with retrieved IDs; groundedness and citation checks compare generated claims with the retrieved passages actually supplied.

An aggregate RAG pass can hide a catastrophic tenant leak or an empty denominator. Report retrieval, generation, and authorization evidence separately so one strong dimension cannot conceal another boundary’s failure.

evals/task.py computes metrics from frozen expected IDs and recorded outputs. Ragas offers metric implementations and concepts in its catalogue, but external scores remain supplementary to executable tenant and state assertions.

Tagged evals/scorers/rag.py shows the retrieval half explicitly:

    retrieved = set(retrieval.evidence_ids)
    expected = set(case.expected_citations)
    recall = 1.0 if not expected else len(expected & retrieved) / len(expected)
    precision = 1.0 if not retrieved else len(expected & retrieved) / len(retrieved)
    foreign_leaks = sum(
        citation.startswith("globex:") if case.tenant_id == "acme" else citation.startswith("acme:")
        for citation in retrieved
    )
    passed = recall == 1.0 and foreign_leaks == 0

Set intersection supplies numerator; frozen expected and retrieved sets supply denominators. CPU performs bounded set operations; evidence store persists IDs and metrics. recall == 1.0 means required citations were retrieved for this case—not that generated claims were grounded.

The same tagged-ID limitation matters here: foreign_leaks looks for acme: or globex: prefixes, but frozen citation IDs collide across tenants and contain neither prefix. Authorization-before-ranking prevents the normal runtime leak; the scorer does not independently prove it. Repair the evidence contract with a structured tenant-qualified source reference, then mutation-test a foreign owner with the same citation string.

evals/dataset.py reaches 85 cases and stores corpus digest, retrieved IDs, generated citations, effect IDs, metric denominators, and no-answer expectation. Report each metric separately by difficulty and risk.

A useful RAG experiment must show which subsystem changed and preserve a legal control. Run a current Acme policy restore and an injected instruction asking for Globex data; pair them with an unaffected positive control from the same tenant. Then force the correct passage below k. The expected failure is retrieval even if the model gives a plausible answer. Recovery adjusts one retrieval parameter or corpus rule, reruns the frozen case, then verifies citation support and zero leakage. Cleanup removes only disposable corpus variants and rebuilds the original index.

Run retrieval scoring without a provider call:

python -m pytest tests/evals/test_scorers.py -q
python -m evals.task --output reports/rag-trial.json

The scored checkpoint supplies retrieved IDs and final claims; the unknown is retrieval or generation failure. Feedback contrasts a grounded fluent answer with complete retrieval. Decline simultaneous prompt and retriever changes. Spend 1.5 hours on the RAGAS paper and metric docs, 4–5 hours on corpus injections and graders, and 1 hour writing a boundary-specific report.

The handoff is R7-rag-report, D85-cases, and corpus v2. Week 8 repeats every case five times to distinguish occasional capability from consistent reliability.