Overview
An agent can produce a convincing answer while doing the wrong thing. It can also produce a different sentence while doing exactly the right thing. Testing its final text alone misses both cases.
My starting recommendation: pytest, a model boundary you can replace, fake tools, and a small versioned set of real-model cases. Use ordinary TDD for deterministic behavior and evaluation-driven development for probabilistic behavior. Add a platform when diagnosing failures or running evaluations has become a measured bottleneck.
This tutorial covers agents as a whole system: the model, its harness (the loop, context, tools, retries, permissions, and budgets), individual steps, and multi-agent workflows. The examples use Python and prioritize open-source frameworks and AWS. Open-source software does not imply free inference, and an open-source framework does not imply an open-weight model.
Read sections 1–3 for deterministic tests, section 4 for executable model evaluations, sections 5–6 for workflow/framework seams, and sections 7–9 for adoption and deployment.
Repository version used here: package 1.2.0, merged in PR #2 at commit e81d83c. The commands below use that implementation. The library PR is already on main; cloning it does not require merging or deploying anything.
What you can clone today: the MIT-licensed Agent TDD Lab supplies the generic runtime, offline tests, a JSON-case evaluation runner, scripted/OpenAI/Bedrock adapters, code graders, comparison reports, a hardened container, and AWS OpenTofu infrastructure. Refund routing is one read-only evaluation example; the separate SQLite refund simulator demonstrates effect safety. You do not need to build every row of the options table.
| Evidence | Supplied by the repository | What your application must establish |
|---|---|---|
| Deterministic behavior | Real harness/workflow tests, scripted decisions, fake tools, mutation check | Your policies, schemas, adapters, and failure cases |
| Model evaluation | Versioned JSON cases, provider adapters, code grading and release gates | Reviewed cases, real-provider runs, quality thresholds and representative traffic |
| Reports | JSON, HTML and JUnit output; repeats and baseline comparison | Review failures and protect artifacts appropriate to your data |
| Deployable job | Non-root container and private AWS Fargate/Step Functions IaC | Your account's IAM, routing, quotas, secrets and staging success |
| Advanced evaluation | Extension seams and deterministic workflow fault tests | Calibrated judges, managed AgentCore evaluation, load/shadow/chaos infrastructure if justified |
Verified at the pinned revision: 193 tests, 100% measured statement/branch coverage, and nine passing CI jobs across the Python/OS matrix, container variants and mocked infrastructure checks. These counts describe that revision; consult current CI for later changes.
Verification has a boundary. Adapter tests against a local HTTP server and AWS SDK stubs exercise real request/response code without calling a hosted model. Mocked OpenTofu tests exercise infrastructure contracts without provisioning AWS. Neither provides live-model quality evidence or confirms a particular AWS account works. Production readiness requires both the reusable engineering foundation and evidence from the deployment that will run it.
1. What exactly are we testing?
Keep the boundaries visible. A model proposes; code controls effects. A refund policy should not depend on whether a model remembers a sentence in a prompt.
| Layer | Failure worth catching | Cheapest useful test | What passing does not establish |
|---|---|---|---|
| Tools and policy | Wrong tenant, invalid arguments, duplicate refund | Pure unit tests and service contract tests | Model chooses the right tool |
| Model adapter | Tool-call parsing, streaming fragments, malformed JSON | Recorded provider fixtures and schema validation | Current provider behavior is unchanged |
| Harness | Infinite loop, missing approval, retry after a committed write | Scripted model decisions through the real loop | Live-model task quality |
| One reasoning step | Bad extraction, unsupported claim, wrong routing | Curated inputs with real model and explicit checks | Entire workflow succeeds |
| Multi-agent workflow | Lost identity, incompatible handoff, duplicate side effects | Scripted workers through real orchestration | Real workers cooperate reliably |
| Whole deployed system | IAM, persistence, timeouts, deployment drift | Small staging end-to-end suite and production observation | Future traffic will match the test set |
Two loops, one specification. A unit test can demand zero unauthorized writes. A model evaluation can estimate how often the agent identifies the right order. Do not average these into a single score that lets eloquent answers hide unauthorized actions.
2. Clone a generic framework, then test one use case
The reusable implementation is Agent TDD Lab on GitHub, under the MIT license. Its core has no refund policy or required vendor SDK. It supports JSON tasks and outputs, injected models and tools, bounded agent loops, and composable workflow steps. This article uses two separate applications of that core: a refund effect simulator for safety invariants and read-only refund routing for model-quality evaluation.
git clone https://github.com/ZGTR/agent-tdd-lab.git
cd agent-tdd-lab
# Reproduce the implementation reviewed in this article.
git checkout e81d83c1579a0b395416fef55141cba04407f720
python -m venv .venv
. .venv/bin/activate
python -m pip install --require-hashes -r requirements-dev.txt
python -m pip install --no-deps --no-build-isolation -e .
python -m pytest
agent-tdd workflow
agent-tdd refund --db demo.db --approve
Python 3.11–3.13 is the tested range. The README also provides a frozen uv.lock path and Windows activation instructions. No API keys, AWS account, GPU, or model service is needed. The core and OpenAI-compatible adapter use the standard library; Bedrock adds the optional AWS SDK. Run without --approve to see an explicit rejection and exit code 2; repeat the approved command against the same database to verify that replay does not create another refund.
The generic core
A Model supplies the next Decision. A Tool implements an application capability. An Agent owns the bounded loop. A Step can be an agent, another workflow, or an ordinary application function. These are structural protocols, so adapters do not have to inherit from a framework base class.
from agent_tdd.adapters.local import ScriptedModel
from agent_tdd.composition import context
from agent_tdd.harness import Agent
from agent_tdd.types import Decision, Status
agent = Agent(
ScriptedModel([Decision("finish", output={"answer": 42})]),
tools={},
)
result = agent.run({"question": "a generic task"}, context())
assert result.status == Status.COMPLETED
assert result.output == {"answer": 42}
context() here is a demo composition helper. Production host code must construct RunContext from verified identity. A completed run means the model produced a valid final decision; it does not establish that its answer is true or that a requested business effect occurred.
Build the refund use case through injection
Use one recurring task: a support agent may refund order A1 for tenant Acme after approval. A second delivery of the same request must not refund twice. Amounts come from the order store, rather than model arguments. The approval is bound to tenant, actor, request, order, and expiry.
The composition root creates RefundTool(store, approvals, policy, clock) and injects it into the generic agent under the name refund. The controller never contains refund-specific rules. This is the test-first boundary:
import sqlite3
from agent_tdd.composition import refund_demo
from agent_tdd.types import Status
def test_missing_approval_creates_no_effect(tmp_path):
database = tmp_path / "refunds.db"
agent, context, store = refund_demo(database, approved=False)
result = agent.run({"request": "Refund A1"}, context)
assert result.status == Status.REJECTED
assert result.error_code == "approval_required"
assert store.pending(100) == []
with sqlite3.connect(database) as connection:
assert connection.execute("SELECT count(*) FROM refunds").fetchone()[0] == 0
Write this test before the approval guard. It should fail when a missing approval permits an effect. Add the guard, rerun it, then prove it detects a regression:
python scripts/verify_mutation.py
That script changes a temporary copy of the policy to invent an approval, runs the relevant test, and requires the intended failure. It does not mutate your working tree. The full suite includes generic harness, workflow, event delivery, and SQLite integration tests; see the repository for the current count and CI results.
Design principles in concrete code
| Principle | Where it applies |
|---|---|
| SRP | Separate domain data, generic harness, workflow, policy, tool controller, SQLite adapter, events, and JSON views |
| DRY | One policy and one effect transaction; the article imports the repository implementation |
| IoC and DI | The composition root injects protocol implementations; no global service locator |
| MVC | Decision/result models, execution/CLI controllers, and JSON rendering views |
| Pub/sub | A topic bus dispatches durable outbox events after the ledger transaction commits |
| OOP and SOLID | Small cohesive objects, structural interfaces, and composition instead of deep inheritance |
| Strategy and ports/adapters | Swap model providers, application steps, graders and storage at explicit interfaces |
| Factory and Composite | StepRegistry creates fresh per-run steps; a workflow composes other steps |
| Decorator and Observer | ValidatedStep checks contracts; injected observers receive lifecycle metadata |
| Unit of Work and transactional outbox | Commit a simulated effect and its event atomically; publish afterward |
The generic workflow passes only declared upstream outputs to a step and shares a trusted context and atomic model-call budget across child agents. Its DAG executes sequentially; fan-out topology does not imply parallel scheduling. For an existing LangGraph, PydanticAI, CrewAI, Strands, or custom harness, implement a Step adapter around that harness. For this library to own the loop, implement its Model and Tool ports instead. Adapter contracts explain both paths.
Do not mock the thing being tested. Replacing the controller with a mock proves little. Replace the provider or external services and execute the actual loop, policy, and orchestration.
3. Eight tests with unusually good return
The refund example exercises these core cases alongside the generic-library tests:
| Case | Required evidence | Why it earns its place |
|---|---|---|
| Missing approval | Explicit blocked result and zero writes | Protects an expensive mistake |
| Order outside tenant scope | Forbidden result and zero writes | Checks the trust boundary |
| Successful task | Correct result and one ledger effect | Prevents a suite that only proves refusal |
| Repeated request | One effect across two deliveries | Models queue and HTTP retries |
| Response lost after commit | Retry returns the result without another effect | Covers an ambiguous distributed failure |
| Repeating model | Loop stops after three calls | Bounds runaway tool cycles |
| Unknown action | Explicit rejection and zero writes | Makes tool dispatch fail closed |
| Key reused for different payload | Conflict error | Prevents silent corruption of deduplication semantics |
The repository example uses durable SQLite, an atomic request key, at-most-one local refund per tenant/order, and a transactional outbox. Tests exercise concurrent duplicate deliveries, payload conflicts, fresh storage instances, rollback when event insertion fails, and a reply lost after commit. It is a local effect simulator: it does not send money. A real payment provider still needs its own stable idempotency key and reconciliation. A local database transaction cannot establish exactly-once behavior across an external service. The event dispatcher is at least once; subscribers must durably deduplicate event IDs.
A step limit bounds turns, not money or wall-clock time. Add provider request timeouts, tool deadlines, cancellation propagation, token limits, concurrency limits, and aggregate budgets across child agents. A spend check after a response measures an overrun; it does not undo the bill. Reserve budget before dispatch and enforce provider-side limits where available.
The highest ROI usually comes from these invariants because they run quickly, explain failures precisely, and remain valid when the prompt or model changes. That is an engineering judgment, not a measured universal ranking.
4. Run evaluations before connecting real effects
Offline tests prove the system handles a proposed action correctly. They cannot prove a model proposes a useful action. The repository now includes an executable evaluation runner: it runs JSON cases through injected steps/providers, applies code graders and explicit gates, and writes machine-readable and human-readable reports.
First run: compare two offline reports
Start without credentials or deployment. Both commands below select the same six public release-partition cases and repeat each three times:
mkdir -p artifacts
agent-tdd evaluate --cases evals/refund-routing.json \
--provider scripted --split release --repeats 3 \
--report artifacts/offline-baseline.json
agent-tdd evaluate --cases evals/refund-routing.json \
--provider scripted --split release --repeats 3 \
--baseline artifacts/offline-baseline.json \
--min-pass-rate 1 --max-regression 0 \
--report artifacts/offline-candidate.json
At the pinned revision, each command passes 18/18 attempts and exits 0. Open artifacts/offline-candidate.html; inspect its .json for individual attempts and slice summaries, or feed .xml to your CI JUnit viewer. A failing evaluation exits 1; invalid configuration or incompatible artifacts exit 2. Archive the baseline before changing the candidate. An identical scripted comparison checks reporting and regression plumbing, not improvement in a model.
The checked-in example asks a read-only router to classify refund requests. It cannot send money or grant approval. Its scripted provider demonstrates runner behavior; passing that run is evidence about the runner and graders, not model quality. The effect simulator in section 2 tests authorization and deduplication independently.
Optional next step: a configured live provider
Before a live run, copy the configuration and replace REPLACE_WITH_YOUR_MODEL_ID with a permitted model ID or Bedrock inference-profile ID. Set the intended region and current input/output prices if you want an estimated-cost gate. Keep credentials in the secret mechanism, outside these files:
cp evals/providers/openai.example.json artifacts/openai.json
cp evals/providers/bedrock.example.json artifacts/bedrock.json
# Edit those copies: model, region/endpoint, token limit and optional prices.
Then select a live provider explicitly when credentials and model access are configured:
# OPENAI_API_KEY must already be available through your secret mechanism.
agent-tdd evaluate --cases evals/refund-routing.json \
--provider openai --config artifacts/openai.json \
--split dev --repeats 3 --report artifacts/openai-dev.json
# Optional AWS SDK dependency; authenticate using the normal AWS credential chain.
python -m pip install --require-hashes -r requirements-bedrock.txt
agent-tdd evaluate --cases evals/refund-routing.json \
--provider bedrock --config artifacts/bedrock.json \
--split dev --repeats 3 --report artifacts/bedrock-dev.json
Read and adapt the checked-in provider configuration to your permitted model, endpoint or region, output-token cap and current pricing. The OpenAI adapter uses the configured full chat-completions endpoint; the Bedrock adapter uses the normal AWS SDK credential chain. Do not commit API keys. The Python evaluator accepts Executor and Grader protocols. Implement run(case, *, request_id, max_requests) -> Sample to wrap an existing agent, Step, harness or workflow; return its normalized RunResult and measured Usage. Honor the supplied request allowance and your provider deadlines. Unknown usage stays unknown rather than being reported as zero. This router is a worked example, not a restriction on the library's domain or framework.
Reuse the evaluator for your own harness
Choose the port that matches the component you already own:
| What you are integrating | Contract to implement | What stays reusable |
|---|---|---|
| A model inside the lab's loop | Model.decide(...); metered models also expose usage() | Harness policy, tools, budgets and AgentExecutor |
| A tool | Tool.execute(...) | Dispatch, trusted context and tool observations |
| An agent, ordinary step or nested workflow | Step.run(task, context) -> RunResult | Workflow, validation and ExecutionService |
| An existing framework/harness or full multi-agent application | Executor.run(case, *, request_id, max_requests) -> Sample | Case loading, repeats, code graders, baseline gates and reports |
| Application-specific quality criteria | Grader.grade(case, sample) -> Grade plus a version | The same runner and reports; hard policy failures remain separate |
This complete Python example uses the supplied generic-agent executor. Run it from the clone after installation; it uses the same case file and writes the same three report formats as the CLI:
from pathlib import Path
from agent_tdd.adapters.local import SystemClock
from agent_tdd.evaluation.cases import load_suite
from agent_tdd.evaluation.executors import AgentExecutor, ScriptedEvaluationModel
from agent_tdd.evaluation.graders import ExactGrader
from agent_tdd.evaluation.reports import write_reports
from agent_tdd.evaluation.runner import evaluate
from agent_tdd.runtime import RunSettings
clock = SystemClock()
executor = AgentExecutor(
ScriptedEvaluationModel,
clock,
settings=RunSettings(max_model_calls=2, timeout_seconds=30),
)
report = evaluate(
load_suite(Path("evals/refund-routing.json")),
executor,
ExactGrader(),
clock,
provenance={"suite_kind": "scripted", "revision": "e81d83c"},
split="release",
repeats=3,
)
write_reports(Path("artifacts/python-example.json"), report)
raise SystemExit(0 if report["passed"] else 1)
To evaluate a LangGraph graph, Strands agent, CrewAI flow, PydanticAI agent or another workflow, replace executor with your application-owned Executor. The runner never imports those frameworks. Create fresh framework state per attempt, pass only case.task to the system under test, preserve trusted identity, normalize output/observations into RunResult, and return measured Usage in Sample. Enforce max_requests at every actual provider call, including child agents and SDK retries. Do not estimate it from the final answer or send case.expected to the agent. The executor integration guide details the adapter boundary; vendor-native execution still needs its own contract tests.
Turn a case set into a release gate
Begin with 12–20 reviewed scenarios: straightforward successes, ambiguous requests, wrong-tenant references, missing approval, retrieval containing hostile instructions, tool failures, and tasks that cannot be completed. The 12 included routing cases—six development and six release—are public instructional examples. Their release label exercises the runner split; it does not make them private or independently held out. Replace and review them for your application, and protect the release fixtures from prompt tuning. Add representative cases for your application, keep development and held-out release splits separate, and version the prompt and graders. Once a held-out failure becomes a prompt example, move it to development and replenish the held-out set.
Here is one case in the runner's actual JSON schema; keep it inside a versioned case-set document. The script is used only by the offline provider. A live model receives the task and tool observations, never the expected answer or script.
{
"schema_version": 1,
"name": "refund-routing",
"version": "1",
"cases": [{
"id": "missing-approval",
"split": "release",
"tags": ["approval"],
"task": {
"request": "Refund order A1",
"order_id": "A1",
"tenant_matches": true,
"known_order": true,
"approved": false
},
"expected": {
"status": "completed",
"output": {"action": "request_approval", "order_id": "A1"},
"max_model_calls": 2,
"forbidden_tools": ["refund"]
},
"script": [{
"action": "finish",
"output": {"action": "request_approval", "order_id": "A1"}
}]
}]
}
In this case, approved is scenario data for a read-only router. It is not an authorization credential. An effectful host must supply independently verified identity and approval.
Reports contain individual attempts and summaries for policy, quality, latency, model calls, tokens and estimated cost. JSON is accompanied by HTML and JUnit reports so people can inspect failures and CI can display them. The supplied reports exclude raw tasks and answers; protect case IDs, tags and other metadata too. Any separate application payload logging needs its own access and redaction policy. Report creation alone does not pass a release: the command must exit successfully after the configured gates pass. Missing attempts, provider errors and grading errors fail closed.
Repeat cases when comparing model versions. Pair baseline and candidate on the same cases and preserve every attempt. Baseline compatibility requires the same cases, split, repeat count, grader and evidence kind; a scripted report cannot stand in for a live baseline. Prompt, model and code may differ: those are the changes the comparison measures. Preserve both versions and their hashes so the comparison is reviewable. Do not retry failed cases until the dashboard turns green.
A tiny suite is a smoke test. Even 20 independent successes with zero failures leave an approximate 95% upper failure bound of 15% using the rule of three. Repeating one case 100 times does not create 100 independent customer scenarios. Choose sample sizes from the failure risk you need to rule out, and report uncertainty and scenario coverage.
Use code graders for schemas, exact routing, amounts, tenant boundaries, citations pointing to retrieved records, and effects. A model judge is an optional extension for semantic criteria such as whether an explanation is useful. Calibrate it against human labels, blind it to candidate identity, version its rubric/model, and audit disagreements. The repository does not claim to supply a calibrated judge. Treat the judge as untrusted input processing too; agent output can contain instructions to the grader.
5. Multi-agent TDD: test the handoffs and the outcome
A planner, researcher, and executor create additional failure boundaries. Give every handoff a schema and trace parent. Carry trusted identity outside model-generated text. A worker cannot grant another worker permissions by writing “approved” in its response.
Start with scripted workers around the real coordinator. Assert that no executor runs before validation; evidence IDs survive the handoff; a worker timeout yields an explicit partial/failure status; a duplicate event cannot duplicate the effect; cancellation stops scheduling new work; and competing workers cannot spend the same reserved budget.
For concurrent branches, test necessary partial order, not one exact transcript. “Validation happened before refund” is a useful contract. “Researcher A always finished before researcher B” is usually scheduler trivia. Exact trajectory matching is appropriate only when the exact sequence is a requirement.
Then run two live layers: each worker independently on its own cases, followed by the full workflow on a smaller suite. This distinguishes a weak worker from a broken handoff. Adding agents can increase cost and failure opportunities; compare against a single-agent baseline before claiming a quality benefit.
6. The same strategy, per framework
The framework changes the injection seam. It does not change the requirements. Keep one domain case set and normalize traces rather than rewriting the definition of success for each framework.
| Setup | Smallest useful test seam | More elaborate option | Likely ROI and adoption trigger |
|---|---|---|---|
| Plain Python | Inject a callable model and tools into the loop | JSON cases plus OpenTelemetry and an evaluation runner | Highest initial simplicity for a small workflow; maintenance rises as orchestration grows |
| PydanticAI | Agent.override with TestModel or FunctionModel; fake dependencies | Pydantic Evals and trace-based checks | Strong for typed outputs; add evaluation tooling when case/report management hurts |
| LangGraph | Inject model/node dependencies; test nodes and compiled graph transitions | Checkpoint/resume tests, trajectory evaluation, optional LangSmith | Strong for branching and durable state; a graph is overhead for a two-step task |
| CrewAI | Test tool functions and Flow state; replace the application-owned worker boundary | Real crew evaluation and role/task-level reports | Useful when role decomposition is justified; live judging alone is a weak policy gate |
| AWS Strands | Inject model/tool boundaries; unit-test policy outside the model | Open-source Strands Evals; graph/swarm trajectory evaluation | Natural AWS-oriented path; keep the first tests credential-free |
| Bedrock / AgentCore | Test application adapter and AWS request contracts | Staging Bedrock calls, AgentCore evaluations and CloudWatch traces | Valuable when AWS runtime/IAM integration matters; highest setup burden for a local prototype |
PydanticAI: model substitution is built in
This minimal example tests the override seam and output plumbing. It does not test tool selection or the refund policy; use FunctionModel to script precise tool calls and run the shared policy cases through your actual agent.
from pydantic_ai import Agent, models
from pydantic_ai.models.test import TestModel
models.ALLOW_MODEL_REQUESTS = False # offline test process only
agent = Agent(TestModel())
def test_output_boundary():
with agent.override(model=TestModel(custom_output_text="approval_required")):
result = agent.run_sync("Refund A1")
assert result.output == "approval_required"
The network guard belongs in your offline test fixture, not your production module. TestModel generates procedural test behavior; FunctionModel gives finer control. Neither measures model intelligence. See the official testing guide.
LangGraph: execute the graph, replace one boundary
This standalone example checks graph wiring and a policy node using injected decisions. A real-model adapter can later replace choose. Add a conditional edge and assert both branches when your graph grows.
from typing import TypedDict
from langgraph.graph import StateGraph, START, END
class State(TypedDict):
approved: bool
action: str
status: str
def build(choose):
graph = StateGraph(State)
graph.add_node("propose", lambda state: {"action": choose(state)})
graph.add_node("gate", lambda state: {
"status": "ready" if state["approved"] else "approval_required"
})
graph.add_edge(START, "propose")
graph.add_edge("propose", "gate")
graph.add_edge("gate", END)
return graph.compile()
def test_graph_gate():
app = build(lambda _: "refund")
result = app.invoke({"approved": False, "action": "", "status": ""})
assert result["status"] == "approval_required"
The repository's generic workflow runner is in-memory; use an appropriate framework when durable orchestration is required. For persistent workflows, test interruption, resume, duplicate delivery, state migration, and distinct thread IDs. An in-memory checkpointer is useful for isolation, but test your real database checkpoint contract separately. See LangGraph testing.
CrewAI: distinguish Flow tests from crew evaluation
Start with a deterministic Flow boundary. This example tests state-to-result wiring without constructing a live LLM. It is deliberately smaller than a full crew.
from pydantic import BaseModel
from crewai.flow.flow import Flow, start
class RefundState(BaseModel):
approved: bool = False
class RefundFlow(Flow[RefundState]):
@start()
def gate(self):
return "ready" if self.state.approved else "approval_required"
def test_flow_gate():
assert RefundFlow().kickoff() == "approval_required"
When a Flow method calls a crew, inject your own worker(task) -> structured_result function at that method boundary; stub that worker in coordinator tests. Then test the real crew separately. CrewAI's test/evaluation features run model-backed assessments and introduce model cost and grader variance. They complement the deterministic suite. See CrewAI testing.
AWS Strands: separate local contracts from cloud evidence
Strands Evals can evaluate an ordinary task function, so it can reuse the offline harness before you enable Bedrock. This is a minimal example of its case/experiment structure; add exact policy checks in pytest and richer evaluators as needed.
from strands_evals import Case, Experiment, eval_task
from strands_evals.evaluators import OutputEvaluator
@eval_task()
def task():
# Replace with your application adapter for a live experiment.
return "approval_required"
def main():
experiment = Experiment(
cases=[Case(name="approval", input="Refund A1", expected_output="approval_required")],
evaluators=[OutputEvaluator(
rubric="Does the response clearly require approval before a refund?"
)],
)
report = experiment.run_evaluations(task)
report.run_display()
# main() # model-backed evaluator: requires configured provider
This rubric judge itself can call a model even though the example task is a constant. It is an API illustration, not a policy test or a runnable offline benchmark. For the high-ROI first pass, keep the lab's exact assertions. The Strands quickstart explains provider setup and experiment execution.
For actual Bedrock agents, exercise one small staging suite with the intended region, model/inference-profile ID, IAM role, request/response adapter, and real timeouts. SDK request stubs can verify parameters and error mapping; they cannot prove model access, IAM authorization, service availability, or task quality.
AgentCore is a managed runtime/evaluation option, not an open-source framework. Its on-demand evaluations analyze selected agent spans. Adopt that layer when you already need AWS trace collection and operational evaluation. Include telemetry ingestion delay, access controls, retention, and evaluation charges in the decision. Do not require it to run a local unit test.
The PydanticAI, LangGraph, and CrewAI examples above passed locally with pydantic-ai-slim==2.57.0, langgraph==1.2.16, and crewai==1.15.28 on Python 3.12. The Strands example was import-checked against strands-agents-evals==1.5.0; its paid judge was not invoked. Framework APIs evolve. Pin and lock the versions you validate in your own project. The repository core is independently executable; the refund use case imports that core. The vendor framework snippets illustrate seams and should be validated against your selected SDK versions before being copied into CI.
7. Options and ROI: earn each additional layer
These are adoption layers, not six deliverables every project must build. Start with A and a small B suite. The repository supplies A, the B runner/provider adapters, and lightweight C reports. D's calibrated judges, E's managed AWS evaluation, and F's operational platforms are optional integrations. Deterministic workflow fault tests already belong in A; you do not need a judge to test a broken handoff.
Deployment is a separate decision. You can clone, run tests, compare offline reports and validate mocked IaC without creating AWS resources. The supplied live-evaluation workflow is manual-only; routine CI does not invoke hosted models or apply infrastructure. The commands in sections 8–9 describe later live/staging work, not a prerequisite for trying the lab.
The following effort ranges estimate adapting a small existing Python service from scratch. Cloning the lab reuses implementation, but does not eliminate case review, integration or live verification. These are planning estimates, not measured benchmarks or vendor quotes. They exclude learning a new cloud platform and large data-cleaning work. Runtime costs depend on your model, token volume, repetitions, tool calls, CI, and retention.
| Option | Estimated initial effort | Ongoing cost | Return and limitation | Upgrade when |
|---|---|---|---|---|
| A. Offline tests — supplied | 2–4 hours | CI minutes; no inference in offline tests | Excellent initial return on policy, retries, budgets; no model-quality evidence | Always add a small live smoke set before release |
| B. Live cases + code graders — runner supplied | 0.5–1 day | Inference per case and repeat | Best general starting balance; catches prompt/model regressions | Reading individual reports becomes slow |
| C. Reporting — lightweight reports supplied; platform optional | 1–3 days | Same inference plus storage/runner maintenance | Better slicing, comparison, and debugging; setup can exceed savings for ten cases | Multiple contributors or frequent model changes |
| D. Calibrated judges + advanced workflow fault simulation — optional | 3–5 days | Extra judge calls, labeling, simulation | Useful for semantic quality and handoff failures; judges can be wrong | Semantic failures dominate manual review |
| E. Managed AWS evaluation — optional; staging contracts required for AWS use | 2–5 days if AWS foundations exist | AWS inference, evaluation, telemetry, storage, CI | Strong operational evidence and integration checks; adds IAM and telemetry work | AWS operations and investigation time justify it |
| F. Chaos/load/shadow/canary platform — optional | 1–2+ weeks | Environments, traffic replay, model/tool spend, maintenance | Valuable for high-impact or concurrent systems; simulated realism has limits | Real incident classes demand evidence cheaper tests cannot provide |
For C, use the supplied reports until a reporting platform earns its upkeep. Pydantic Evals and Strands Evals are possible upgrades; other open-source options include DeepEval, promptfoo, and Arize Phoenix. Compare their current licenses, local execution, telemetry defaults, and required integrations before choosing. These are alternatives with overlapping capabilities, not a stack you need to install together.
Put numbers on the decision
Use a fixed horizon, such as one month. Count saved investigation time and expected avoided incident loss without double-counting the same benefit.
Monthly benefit = debugging hours saved × hourly cost
+ expected incident loss avoided
Monthly cost = setup amortization + maintenance + inference + infrastructure
ROI = (benefit - cost) / cost
Illustrative assumptions: labor costs $80/hour; setup is amortized over three months. You estimate that a minimal suite saves three debugging hours and avoids $100 of incident loss per month. These are your estimates to challenge, not promises of causal savings.
| Illustrative option | Monthly benefit | Monthly cost | Monthly net | ROI |
|---|---|---|---|---|
| Minimal: 4h setup, 1h upkeep, $10 runtime | $340 | $196.67 | $143.33 | 73% |
| Reporting layer: 12h setup, 2h upkeep, $60 runtime; saves 6h + $200 loss | $680 | $540 | $140 | 26% |
| Operational suite: 30h setup, 4h upkeep, $200 runtime; saves 12h + $600 loss | $1,560 | $1,320 | $240 | 18% |
These are three total-system scenarios, not incremental upgrades. To decide whether to move from one to another, compare incremental benefit against incremental cost. In this example reporting increases monthly cost by $343.33 and benefit by $340, so it does not yet earn its upgrade on financial grounds. Other requirements can still justify it.
For inference budgeting, count both candidate and baseline, every repeat, and judges. Example assumptions: 20 cases × 3 repeats × 2 versions = 120 agent runs. At an assumed $0.02 per complete run that is $2.40; one $0.005 judge call per run adds $0.60. The $3 total excludes tools, CI, telemetry, retries, and labor. Those unit costs are illustrative, not current Bedrock or model prices. Multi-agent cost is the sum of all worker/model/tool calls, not just the final response.
8. Make release evidence executable
Put deterministic tests on every change. Run a small live evaluation when prompts, models, tools, retrieval, or orchestration change, followed by a broader held-out comparison before release. Keep credential-bearing evaluation jobs separate from untrusted pull-request code. An unknown provider or absent credential must fail; never silently substitute the scripted provider.
# Offline gate: mandatory and cheap.
python -m pytest --cov
python scripts/verify_mutation.py
# Baseline: same cases, split, repeats and grader as candidate.
agent-tdd evaluate --cases evals/refund-routing.json \
--provider openai --config artifacts/openai.json \
--split release --repeats 3 --report artifacts/baseline.json
# After selecting the candidate model in a reviewed config:
agent-tdd evaluate --cases evals/refund-routing.json \
--provider openai --config artifacts/openai.json \
--split release --repeats 3 --baseline artifacts/baseline.json \
--min-pass-rate 1 --max-regression 0 --max-model-calls 2 \
--max-requests 100 --timeout 30 --max-latency 30 \
--max-total-tokens 20000 --max-estimated-cost-usd 1 \
--report artifacts/candidate.json
The cost gate requires configured input/output prices and reported token usage; missing measurements fail closed. The thresholds above are an example policy for a small routing smoke set; size them to the case count and risk. Archive the reviewed baseline before changing candidate configuration. A baseline and candidate made from identical configuration can check the comparison plumbing; they do not establish an improvement. Estimated cost depends on configured prices and reported token usage; reconcile it with provider billing. Token and estimated-cost thresholds are release gates checked after each case run, not hard billing caps. One case can make multiple model requests and overrun its token or cost threshold before the runner blocks later cases. A spend measurement cannot undo a bill. The runner bounds dispatch with per-run model calls, a global request cap, request timeouts, input size and configured output-token caps; apply provider-side spend quotas where available.
Require zero policy violations, an explicit task-quality floor, bounded regression, and latency/call/token/cost limits appropriate to the application. Do not average a policy violation into a quality score. Unexpected errors fail the gate. CI for the public repository exercises offline scenarios and stubbed provider contracts without exposing hosted-model credentials. The manual Live evaluations workflow runs only from main: configure its live-evaluation environment with OPENAI_API_KEY or the intended AWS OIDC role/account, then select the provider and permitted model. It allows up to 12 inference requests for the candidate and another 12 when requesting a baseline; those calls can incur charges. Its reports establish evidence only about the selected public smoke cases. Your trusted application release job must evaluate representative, protected fixtures. GitHub environment creation requires repository administration rights. An administrator can configure and verify its main-only branch policy with the supplied scripts/configure_live_environment.py setup tool; it accepts no secret values and refuses to widen a different existing policy. The connection used to verify this revision received 403 for environment administration, so no environment/secret/OIDC setup or live run was completed here.
Record enough to reproduce a run: code commit, prompt hash, model/provider ID and settings, tool schemas, retrieval snapshot, framework lockfile, case-set version, grader version, time, seed when supported, and trace IDs. Preserve application-specific provenance alongside the report when it is outside the runner's metadata. Temperature zero and a seed can reduce variation; neither guarantees deterministic hosted inference.
9. Deploy the job, then verify the environment
The deployment runbook includes a hardened container and OpenTofu AWS root configuration. The library also works without Docker or AWS. Its default container runs the scripted demonstration as an on-demand job, with no HTTP endpoint or always-running service. To deploy your agent, build an application composition root that registers your own step factories and injects trusted identity, providers and tools.
from agent_tdd.adapters.local import SystemClock
from agent_tdd.composition import research_workflow
from agent_tdd.runtime import ExecutionService, StepRegistry
registry = StepRegistry({"research": research_workflow})
service = ExecutionService(registry, SystemClock())
# Host code calls service.run(...) with verified identity and a stable request ID.
Factories create fresh steps per run. Validators wrap steps without coupling the core to one schema library; observers receive lifecycle metadata without raw tasks, outputs or provider exception text. These patterns make extensions explicit. They do not require a plugin loader, inheritance hierarchy or new framework for each use case.
The IaC creates ECR, ECS, separate execution/application/orchestration roles, no-ingress networking, log retention, and—after supplying an image digest—a Fargate task and Step Functions runner. The task runs as a numeric non-root user with a read-only root filesystem and dropped capabilities. The application role starts with no permissions. No task starts during infrastructure apply, and the state machine has no automatic effect retries.
Validate the configuration without AWS credentials or resource creation:
python scripts/install_tofu.py --directory /tmp/agent-tdd-tools
export PATH="/tmp/agent-tdd-tools:$PATH"
tofu -chdir=infra/aws init -backend=false -lockfile=readonly
tofu -chdir=infra/aws fmt -check -recursive
tofu -chdir=infra/aws validate
tofu -chdir=infra/aws test
The checksum-pinned installer supports Linux/amd64; use OpenTofu's official installer elsewhere. For an actual deployment, follow the runbook's complete sequence: configure an existing encrypted/locked S3 state backend and private VPC connectivity; review/apply a bootstrap plan; build/push the image; set the immutable digest; review/apply the runner plan; start an execution and require SUCCEEDED. Add application-specific permissions, provider secrets and durable services explicitly. These are environment-dependent operations and can incur AWS charges.
Before enabling real effects, verify authenticated host identity, authoritative approvals, timeouts, live-provider access, deployed IAM/network paths, durable effect idempotency, reconciliation, cancellation, alarms and rollback. The SQLite simulator stays local: Fargate storage is ephemeral, and SQLite WAL on shared EFS/NFS is not a replacement for a remote effect store. Step Functions cancellation is best effort; monitor and reconcile orphaned work.
The repository supplies tested building blocks and a deployment path. It does not certify arbitrary agents, prompts or accounts. Its verification covers offline tests, provider HTTP/SDK contract tests, container execution and mocked infrastructure checks. Live-model runs, provisioned AWS integration and a particular application's effectful contracts must be demonstrated in that application's environment. Follow the production integration requirements, then turn each production failure into a minimized regression at the cheapest layer that reproduces it.