
How to Test an AI Agent Before Your Whole Team Relies on It
A demo shows that an agent can work once. A test shows how it behaves across the cases your team handles every week, before anyone depends on it for real work.
A small team can do it without an evaluation platform: write down what success means, save 20 to 50 real cases, run each one more than once and pilot it slowly.
Anthropic and OpenAI publish how they test AI systems, and a 2024 benchmark paper measured how fast an agent loses consistency when the same task is repeated.
Define success before you look at answers
Write down what a correct result must contain before the first run. Criteria written afterward can end up describing whatever the agent produced, and a confident answer can pass on tone alone.
Write criteria a stranger could check
Anthropic's documentation on success criteria and evaluations asks for criteria that are specific, measurable, achievable and relevant. Its bad example, "the model should classify sentiments well", becomes a good one when it names a score, a test set and the baseline it improves on. Anthropic's engineering guide adds a practical bar: a good task is one where two domain experts would independently reach the same pass/fail verdict.
For a supplier-research agent, that might mean a finding for every requirement, a source for each finding, unknowns marked as unknown and no contact with vendors. Each item can be checked by someone who has never seen the agent, regardless of how confident the prose sounds.
What a written test case looks like
Anthropic's guide to agent evaluations includes a theoretical example test for a support agent that must handle a refund for a frustrated customer. It grades the case four ways: a rubric for the reply, a check that the ticket ended up resolved and the refund processed, a list of tool calls that must happen, from verifying the customer's identity to sending a confirmation, and a cap on the conversation's length. The refund call passes only with an amount of 100 or less, so a case can fail on the path even when the customer ends up happy.

That cap is 10 turns. Each line is something a person or a short script can confirm after the run, and none of it depends on how polished the agent's wording was. Beyond whether the case passes, Anthropic's example also tracks how many turns, tool calls and tokens the run took, and its latency, such as the time to the first token.
Check the path as well as the result
The guide separates the task, the transcript of what the agent did and the outcome, which it defines as the final state of the environment at the end of the run. Its example: a booking agent may say the flight is booked, but the outcome is whether a reservation exists in the database. An agent that reaches the right outcome by doing something it should not is still failing.
Build a test set from real work
The first cases should come from work your team already did, with sensitive details removed. Then add the inputs that break things, plus cases where the agent should decline or do nothing at all.
Start with 20 to 50 real cases
Anthropic's engineering guide says that 20 to 50 simple tasks drawn from real failures is a great start, because early changes to an agent often have large, visible effects that small samples can detect. OpenAI's evaluation guide makes a related point: log everything as you develop, so you can mine the logs for good test cases.
A month of emails, tickets or requests the agent will handle is a practical source. Keep the original input, what a person did with it and whether that answer was right.
Add the cases that break things
Anthropic's documentation lists edge cases worth including: irrelevant or missing input data, overly long input, poor or harmful user input and ambiguous cases where even people would struggle to agree. OpenAI's guide asks for typical, edge and adversarial cases in the same set. For a small business, that adds up to five groups:
- Ordinary cases: the requests the agent will see week to week.
- Missing or conflicting information: one required input absent, or two sources that disagree.
- Out-of-scope requests: asks it should refuse or hand to a person.
- Tool failures: a system that returns an error or nothing at all.
- Hostile content: a document or email with instructions meant to redirect the agent. Our prompt injection post covers how to plant harmless test instructions.
Ask the same question several ways
Anthropic's consistency example for an FAQ bot groups several versions of one question: a return-policy question with typos, a long rambling version and one padded with irrelevant details about a cousin. All of them should get the same answer. Customers can ask in any of these ways, so a test set with only clean wording overstates how the agent will do.
Test what it should not do
The guide warns that one-sided evals create one-sided optimization: test only that an agent searches when it should, and you may end up with one that searches for almost everything. Include cases where the right move is to stop, ask or hand over. An honest "I could not complete this" counts as a correct result when the task cannot be done within the rules.
Write a reference answer
For each case, the guide recommends a reference solution, a known working output that passes every check. It proves the case can be solved and that the checks are set up correctly. If your own team cannot produce a passing answer, the case needs rewriting before it can judge the agent.
Record enough to explain a failure
A score without a record tells you something went wrong and nothing about where. Save enough from each run that a person can retrace it later and decide what to change in the instructions or tools.
What to save for each run
For every run, save the case, the instructions version, the tools available, the full transcript, the final output and the reviewer's verdict. The guide defines the transcript as the complete record of a run, including outputs, tool calls, reasoning and intermediate results. A shared spreadsheet with one row per run and a link to each transcript is enough at this stage.
Score correctness and presentation separately
A well-formatted table with an unsupported claim fails. A correct answer in an unusable format also fails, for a different reason with a different fix. The guide describes three ways to combine checks on one case: weighted scores that must reach a threshold, a binary rule where every check must pass, or a hybrid. For customer-facing work, the binary rule is the simpler one to explain to the team.
Three ways to grade results
Anthropic describes three kinds of graders, each with a different cost and blind spot. A small business test set can use all three, each where it is strongest, and check one against another.
Code checks
Anthropic's documentation describes code-based grading as fast, reliable and extremely scalable, but lacking nuance for complex judgments. Use it for anything objective: required fields are present, the output follows the template, no email was sent, the CRM record is unchanged. The engineering guide notes that code checks are brittle to valid answers that do not match the expected pattern exactly.
Model graders
A second AI scores the output against a written rubric. The documentation calls this fast, flexible and suitable for complex judgment, with one condition: test to ensure reliability first, then scale. Two habits from the engineering guide help: let the grader answer "Unknown" when it lacks information, and grade each criterion with a separate grader call in place of one overall verdict. The documentation also asks for detailed rubrics and a fixed output, such as "correct" or "incorrect" or a score from 1 to 5.
Human review
The engineering guide calls human grading the gold standard and the way to calibrate model graders. The documentation, written for testing prompts and LLM applications, calls it slow and expensive and says to avoid it if possible. Both points hold for a small team: a person reviews the high-stakes cases and a sample of the model grader's verdicts, and OpenAI's guide names the same habit, using human feedback to calibrate automated scoring.
Measure consistency, not just success
An agent can answer the same input differently, so one passing run proves little. For work a customer will see, count a case as passed only when every repeated run of it passes.

Two ways to count repeated runs
Anthropic's guide distinguishes pass@k, whether at least one of k attempts succeeded, from pass^k, whether all k attempts succeeded. Pass@k suits work where a person picks one of several drafts. Pass^k is the measure for anything a customer will see, and the guide notes that the two diverge as the number of runs grows.
The gap is larger than it sounds. In Anthropic's example, an agent that succeeds 75% of the time on a case will pass all three of three runs only about 42% of the time (0.75 × 0.75 × 0.75). For important cases, run each one three times and count it as passed only when every run passes.
What a published benchmark found
The pass^k measure comes from τ-bench, a 2024 benchmark paper that simulates customer conversations for retail and airline agents and checks the database state at the end of each one. The paper's top function-calling agent, built on gpt-4o, solved 61% of retail tasks and 35% of airline tasks on a single run. Counted as passed only when all eight of eight runs succeeded, its retail score fell below 25%.
Compare with how the work is done today
A pass rate means little on its own. Set it beside the process the agent would replace, measured on the same saved cases, and judge the result by what each kind of error would cost you.
What to measure on both sides
| Measure | Why it matters |
|---|---|
| Share of cases completed correctly | The basic usefulness test |
| Time to a usable result | Where the savings come from |
| Correction time per case | A fast answer that needs rework saves nothing |
| Boundary violations | Even one may be disqualifying |
| Cost per case | Usage fees must stay below the value of the time saved |
Anthropic's documentation includes latency and price in its list of common success criteria, next to task fidelity, consistency, tone and privacy. Record them per case, since an answer that takes a person ten minutes to fix may cost more than doing the work the current way.
Set the bar by the cost of an error
A typo in an internal summary and a refund sent to the wrong customer do not belong in the same average. Set a separate threshold for each kind of error, and treat a boundary violation, such as contacting a customer without approval, as a failed test whatever the overall score.
Read the transcripts behind the scores
A score is a claim about the agent that nobody has checked until someone reads the runs behind it. Sometimes the agent made the mistake, and sometimes the test case or the grader did.
When the test itself is wrong
Anthropic's guide gives an example from its own work: Claude Opus 4.5's score on a benchmark called CORE-Bench, before the grader was fixed, was 42%. An Anthropic researcher then found rigid grading, ambiguous task specifications and tasks that could not be reproduced exactly. The guide reports that the score jumped to 95% after fixing bugs and using a less constrained scaffold, the setup the agent runs in. The rigid grading penalized "96.12" when it expected "96.124991…".

The same mistakes fit in a small test set: a check that wants an exact phrase, a case with two defensible answers or a reference answer that was itself wrong. Each one makes the agent look worse or better than it is.
Read every failure
The guide says that you cannot tell whether your graders work until you read the transcripts and grades from repeated trials. Read the transcript of every failed run, plus a sample of passes, and mark whether the fault lies with the agent, the case or the grader. OpenAI's guide lists the habit to avoid by name: vibe-based evals, judging from the impression that it seems to be working.
Keep two test sets as the agent matures
Once the agent works, the test set has two jobs: pushing what the agent can do and protecting what it already does. Anthropic's guide keeps a separate set for each job.
Capability tests
Capability tests ask what the agent can do well. The guide says they should start at a low pass rate, aimed at tasks the agent struggles with, so the team has room to improve. New kinds of work for the agent belong here first.
Regression tests
Regression tests ask whether the agent still handles everything it handled before, and the guide says they should pass at nearly 100%. When a capability case passes reliably, move it into the regression set. OpenAI's guide recommends running evals on every change and growing the set over time, which for a small team means rerunning the regression set after each edit to instructions, tools or model.
Roll out in stages
Saved cases show how the agent handles the past. A staged rollout shows how it handles the present, with a person checking its work while the stakes are still small and every result is reversible.
A pilot with every action approved
Move from saved cases to a pilot with one person or one queue, with approvals on everything the agent sends or changes. Keep the run records from the pilot in the same format as the test set, so a failure can become a new saved case the same day.
Widen use when results hold
Review the log weekly. Widen use only when results hold across the pilot and the regression set still passes, and relax one approval at a time, starting with the actions that are easiest to undo.
Example: a support-inbox agent
Picture a three-person online shop testing an agent that drafts replies to support emails. The owner saves 30 past emails, including two with no order number and one asking for a refund outside the policy. Each case runs three times. For the pilot, the office manager approves every draft for two weeks, and any draft she rewrites becomes a new case.
FAQs About Testing AI Agents
Short answers on the size of a first test set, whether an AI can grade its own work, when to retest and whether a small team needs special software before it starts testing.
How many test cases do I need?
Anthropic's engineering guide calls 20 to 50 simple tasks drawn from real failures a great start. Include the awkward cases, and add every real failure you find later to the set.
Can the AI grade its own work?
A model grader can check format and clear errors at scale. Anthropic's documentation says to test it for reliability first, so a person should judge important cases and compare a sample of its verdicts with their own.
When should we retest?
After any change to instructions, tools, data sources or the model, and on a schedule even without changes, since connected systems change too.
Do we need an evaluation platform?
Not to start. A spreadsheet of cases, verdicts and links to transcripts covers a first test set. If you choose a platform, check its future: when we checked on November 16, 2026, OpenAI's guide said its Evals platform had become read-only for existing users on October 31, 2026 and was scheduled to shut down on November 30, 2026.
Test before you trust
Written criteria, 20 to 50 saved cases, three runs for each important case, graders you have checked and a pilot with approvals will show whether the agent is ready for real work.
Every agent we deliver comes with a test set built from your cases and a staged rollout; see what is included in our AI agents and skills service. If you already have an agent that behaves unpredictably, tell us what the agent is supposed to do, and we will suggest the first ten test cases.
Last checked: November 16, 2026, against Anthropic's engineering guide to agent evaluations (January 9, 2026) and its documentation on success criteria and evaluations, OpenAI's evaluation best practices guide and the τ-bench paper (arXiv 2406.12045). Vendor documentation changes; check the linked pages before relying on a detail.
Evaluation methods and figures come from Anthropic, OpenAI and the τ-bench paper, checked November 16, 2026.



