Eval Engineering: Why Agent Evaluation Is Different

When we started building AI agents at Orioverse, we made the same mistake everyone makes. We evaluated our agents using standard NLP metrics. Accuracy, precision, recall. The numbers looked great. The agents were failing in production.

The problem is fundamental. Model evaluation measures outputs against a fixed ground truth. Agent evaluation measures decisions against an unbounded possibility space. These are different problems requiring different tools.

The Three Grader Types

Anthropic's engineering team identified three types of graders that work together in effective eval systems. We have confirmed this pattern across our own work at Orioverse and in conversations with teams building production agents.

Code-Based Graders

Code-based graders are the simplest and most reliable. They check deterministic properties using regular code. Did the tool call include the required parameters? Does the output match the expected format? Is the response within acceptable length?

These graders are fast, cheap, and deterministic. They never hallucinate. Every agent eval suite should start here. OpenAI's Evals framework and the open-source Harbor project both support code-based graders.

The limitation: code graders only catch what you can express in code. They miss semantic problems. The tool call was technically correct but logically wrong.

Model-Based Graders

Model-based graders use one LLM to evaluate another. The evaluator model reviews the agent transcript and scores each decision. Did the agent choose the right tool? Was the reasoning sound? Did it handle errors gracefully?

This sounds recursive but it works well. In our testing, a fast model like Claude Haiku can review a transcript and score each decision in about 200ms per turn, at a fraction of a cent per evaluation.

The key trick: the evaluator model needs a clear rubric. Without structured criteria, model-based graders drift. Providing explicit scoring criteria with examples of passing and failing behavior dramatically improves grader reliability.

OpenAI's Evals framework supports model-based graders natively. Tools like Braintrust and LangSmith also provide model-graded eval capabilities.

Human Graders

Human graders are the most expensive but also the most reliable for subtle problems. You sample a subset of agent interactions and have a person review them against the same rubric.

The art is in the sampling strategy. Review 100% of interactions involving money or sensitive data. Review 100% of interactions the model-based grader flagged as borderline. Review a random 5% of everything else.

Human review is not just for catching failures. It also calibrates model-based graders. Every time a human disagrees with a model grader, that disagreement is a signal that your rubric needs refinement.

Capability Evals vs Regression Evals

A distinction that changed how we think about evaluation: capability evals measure what your agent can do, while regression evals measure whether changes broke something that used to work.

Capability evals are benchmarks. They compare your agent to other agents or to previous versions. SWE-Bench Verified is a capability eval for coding agents. TAU-bench evaluates task-oriented agents. These are useful for understanding the ceiling of your system.

Regression evals are tests. They replay specific scenarios and check that the agent still makes the right decisions. These are far more valuable for day-to-day development because they catch regressions before they reach users.

The corpus approach: maintain a set of canonical agent interactions, each annotated with the correct decisions at each turn. Before every deployment, replay these interactions and flag any divergence. The corpus grows organically -- every production interaction that required human intervention becomes a regression test case.

This is the pattern used by teams at Anthropic, OpenAI, and every production agent deployment we have studied.

Eval Saturation

One problem that emerging teams do not expect: eval saturation happens when your agent passes all test cases with no room for improvement. At that point, your eval stops providing signal.

Anthropic's engineering blog describes this phenomenon. SWE-Bench Verified scores started around 30% and frontier models are now nearing saturation above 80%. As evals approach saturation, large capability improvements appear as small increases in scores.

The fix is to continuously refresh your eval corpus. Add harder cases. Add edge cases from production. Add cases that specifically target known failure modes.

The Eval Roadmap

Anthropic recommends a phased approach to building agent evals, and we follow the same pattern at Orioverse:

Phase 1: Define success criteria for your agent. What does a good interaction look like? What does a bad one look like? Write these down before you write any eval code.

Phase 2: Build code-based graders for the deterministic checks. These are fast to build and catch the most common failures.

Phase 3: Add model-based graders for the semantic checks. Iterate on the rubric until grader agreement with human reviewers reaches 90% or higher.

Phase 4: Set up a production monitoring pipeline that samples interactions and runs them through your eval suite continuously.

Phase 5: Use the disagreements between graders to refine your rubrics and grow your regression corpus.

This is not a one-time setup. It is a continuous process that evolves with your agent.

Tools and Frameworks

Several open-source and commercial frameworks can help teams implement agent evaluations. The right choice depends on your agent type, existing stack, and whether you need offline evaluation or production observability.

OpenAI Evals is the most widely used open-source framework. Its 2026 release added first-class support for multi-turn agent traces, tool-call grading, and trajectory quality evaluation.

Harbor is designed for running agents in containerized environments with infrastructure for running trials at scale across cloud providers.

Braintrust combines offline evaluation with production observability and experiment tracking.

LangSmith offers tracing, offline and online evaluations, and dataset management with tight LangChain integration.

Langfuse provides similar capabilities as a self-hosted open-source alternative.

Many teams combine multiple tools or start with simple evaluation scripts. The framework matters less than the quality of your test cases and graders.

Read the full Context Engineering series: orioverse.com/blog/what-is-context-engineering

Eval Engineering: How We Evaluate AI Agents at Orioverse