Agents are difficult to evaluate for the same reasons they are useful: they act over many turns, choose tools, modify an environment, and adapt to intermediate results. Anthropic’s field guide reframes evaluation as a living product discipline built around realistic tasks, mixed grading methods, and careful review of what actually happened.
01
The anatomy of an agent evaluation
A useful evaluation separates the task, each attempted trial, the graders, the recorded transcript, and the final environment state. The transcript explains the path an agent took; the outcome establishes what changed. A support agent can claim that a refund was issued, for example, while a database check reveals whether the refund actually exists.
The agent harness also belongs inside the object being evaluated. Tool definitions, prompts, context management, and orchestration can change performance even when the underlying model remains the same. Scores should therefore describe a complete system and its environment, not a model in isolation.
02
Use the right grader for each claim
Code-based graders are fast, inexpensive, and reproducible. They are well suited to tests, state checks, static analysis, and clear constraints. Model-based graders can assess open-ended work against a rubric, but they are probabilistic and must be calibrated. Human experts remain the strongest reference for nuanced quality, though their time and cost limit scale.
Effective suites combine these approaches. Deterministic checks should verify objective outcomes; model judges should handle flexible or qualitative criteria; and humans should validate the system, adjudicate difficult cases, and periodically recalibrate automated judgments.
- Prefer outcome checks when success creates a verifiable state change.
- Use focused rubrics instead of asking one judge to assess every dimension at once.
- Avoid rigidly prescribing a tool-call sequence when multiple valid solutions exist.
- Allow partial credit for tasks with several independently valuable components.
03
Capability and regression suites answer different questions
Capability evaluations ask what the agent might learn to do next. They should contain hard, discriminating tasks and begin with room for improvement. Regression evaluations protect behavior the product already handles reliably and should remain near a full pass rate.
As a capability suite saturates, its successful tasks can graduate into the regression bank. Maintaining both prevents teams from mistaking stability for progress, or gaining new abilities while silently breaking established ones.
04
Reliability requires more than one run
Agent behavior varies between trials, so a single pass or failure is weak evidence. Pass@k asks whether at least one of several attempts succeeds, which is useful when the product can generate and select among alternatives. Pass^k asks whether every attempt succeeds, which better reflects experiences where users expect dependable first-time completion.
The two metrics diverge rapidly as the number of trials grows. Teams should select the metric that matches the product promise and report the trial setup alongside the result.
05
An eval-driven development roadmap
A credible program can begin with 20 to 50 tasks drawn from manual tests, bug reports, and real user failures. Tasks need unambiguous instructions and a known reference solution that proves both the problem and the grader are valid. Balanced datasets must also include cases where a behavior should not occur.
Trials should begin from isolated, stable environments so cached data, leftover files, or resource contention do not distort results. Teams then need to read transcripts regularly, inspect surprising grades, watch for saturation, and assign clear ownership for keeping the suite healthy.
06
Evals are necessary, but not sufficient
Automated suites provide repeatability and scale before deployment. They cannot fully replace production monitoring, user feedback, controlled experiments, manual transcript review, and systematic human evaluation. Each method exposes a different class of failure.
The strongest operating model treats evaluation as a shared language between product, engineering, research, and domain experts. The suite becomes both a release gate and a precise definition of what the agent is supposed to accomplish.
Alpheva perspective
What we take from the work
- Define the task, transcript, environment outcome, graders, and harness separately.
- Combine deterministic, model-based, and human grading rather than forcing one method everywhere.
- Measure repeated performance with a reliability metric aligned to the product experience.
- Continuously inspect transcripts and repair the evaluation itself when valid solutions are penalized.