Eval Engineering Infrastructure: Building Agent Evals Into Your Dev Loop
Eval Engineering Infrastructure: Building Evals Into Your Development Loop
Anthropic's engineering team published the most practical guide we have seen on building agent evaluations. One number stood out: you only need 20 to 50 simple eval tasks drawn from real failures to start catching meaningful regressions. Teams delay building evals because they think they need hundreds of tasks. They do not.
This post covers the infrastructure that makes agent evaluations useful in a real engineering workflow.
Eval-Driven Development
Anthropic recommends a practice they call eval-driven development: build eval tasks to define planned capabilities before agents can fulfill them, then iterate until the agent performs well. Internally, Anthropic builds features that work well enough today but are bets on what models can do in a few months. Capability evals that start at a low pass rate make this visible. When a new model ships, running the suite immediately reveals which bets paid off.
This reverses the usual workflow. Instead of building an agent and then writing tests for it, you write the tests first to define what success looks like. The agent iterates against the tests. This forces concrete thinking about success criteria before any code is written.
The Eval Harness
An evaluation harness is the infrastructure that runs evals end-to-end. It provides instructions and tools, runs tasks concurrently, records all the steps, grades outputs, and aggregates results.
Anthropic's guidance on building eval harnesses emphasizes environment isolation above everything else. Each trial should start from a clean environment. Unnecessary shared state between runs (leftover files, cached data, resource exhaustion) causes correlated failures due to infrastructure flakiness rather than agent performance. In some internal evals, Anthropic observed Claude gaining an unfair advantage on some tasks by examining the git history from previous trials.
The agent in the eval must function roughly the same as the agent used in production. An agent harness (or scaffold) is the system that enables a model to act as an agent: it processes inputs, orchestrates tool calls, and returns results. When you evaluate an agent, you are evaluating the harness and the model working together.
Infrastructure Noise
Anthropic published a dedicated study on how infrastructure configuration affects agent benchmark scores. The finding was striking: resource allocation can swing scores by over 6 percentage points - more than the gap between top models on some leaderboards.
In their experiments on Terminal-Bench 2.0, strict resource enforcement (where containers were killed the moment they exceeded limits) caused a 5.8% infrastructure error rate. Moving to uncapped resources dropped this to 0.5%. The success rate increased by 6 percentage points between strict enforcement and uncapped allocations.
The practical implications: every element of the evaluation setup can influence the final score, from cluster health to hardware specs to concurrency level. Agentic evals are end-to-end system tests by construction, and any component of that system can act as a confounder. Anthropic recommends that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched.
CI Integration
An effective eval pipeline triggers on every pull request - not a nightly batch. When a developer changes agent behavior, the pipeline replays the regression corpus against the proposed changes and reports regressions inline on the PR.
Anthropic's approach combines several evaluation methods that map to different stages of agent development. Automated evals run in CI/CD on every agent change and model upgrade as the first line of defense against quality problems. Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures. A/B testing validates significant changes once you have sufficient traffic. Systematic human studies calibrate LLM graders and evaluate subjective outputs.
The most effective teams combine these methods: automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.
Eval Maintenance
Maintaining eval suites long-term requires organizational structure. At Anthropic, what proved most effective was establishing dedicated evals teams to own the core infrastructure, while domain experts and product teams contribute most eval tasks and run the evaluations themselves.
Anthropic also invested in tooling for viewing eval transcripts and regularly takes the time to read them. When a task fails, the transcript reveals whether the agent made a genuine mistake or whether the graders rejected a valid solution. Reading transcripts surfaces key details about agent and eval behavior that no automated score can capture.
Tools and Frameworks
Several frameworks can help teams implement agent evaluations. The right choice depends on your agent type, existing stack, and whether you need offline evaluation or production observability.
Harbor is designed for running agents in containerized environments, with infrastructure for running trials at scale across cloud providers and a standardized format for defining tasks and graders. Popular benchmarks like Terminal-Bench 2.0 ship through the Harbor registry.
Braintrust combines offline evaluation with production observability and experiment tracking. Its autoevals library includes pre-built scorers for factuality, relevance, and other common dimensions.
LangSmith offers tracing, offline and online evaluations, and dataset management with tight LangChain integration.
Langfuse provides similar capabilities as a self-hosted open-source alternative for teams with data residency requirements.
Arize offers Phoenix, an open-source platform for LLM tracing, debugging, and offline or online evaluations, and AX, a SaaS extension for scale and monitoring.
Many teams combine multiple tools. The framework matters less than the quality of the eval tasks and graders you run through it.
Getting Started
Anthropic's phased approach: start with the manual checks you already run during development - behaviors you verify before each release and common tasks end users try. If you are already in production, look at your bug tracker and support queue. Converting user-reported failures into test cases ensures your suite reflects actual usage. Prioritizing by user impact helps you invest effort where it counts.
Read the full Context Engineering series: orioverse.com/blog/what-is-context-engineering