An AI agent can score better overall and still become worse at the customer task that matters most. More helpful explanations might lift the average while a change breaks the way it handles a revised refund request.
For a founder approving a release, the useful evidence is whether the agent completes important journeys within the rules the business has agreed. That includes what the customer asks along the way, which actions the agent takes and what actually changes in the system.
AI agent evaluation means defining those expectations and checking runs against them. I would start with one complete task that the team can inspect from beginning to end.
Start with a customer who changes their mind
Consider this fictional support scenario. The order, prices and policy below are invented for the example; this is not a client case or a tested agent.
An authenticated customer has access to an order they paid £80 for: a £60 lamp and a £20 mug. Neither item has been refunded. Under this fictional store's policy, both are eligible for an immediate refund to the original payment method. There are no additional charges. The assistant must confirm the current items and total with the customer before submitting a refund.
The customer initially asks to refund the whole order. The assistant prepares an £80 refund and asks for confirmation. Before any refund is submitted, the customer changes the request:
Actually, I'll keep the mug. Refund only the lamp.
The assistant should now ask the customer to confirm a £60 refund for the lamp. After the customer confirms, it should submit that refund and report the tool's result accurately.
This is the point where an evaluation of the final reply alone can mislead. The assistant might write a convincing confirmation while submitting the earlier £80 amount. It might refund £60 against the wrong item, or claim completion after the refund service returned an error.
Write the expected result before experimenting with prompts: one completed £60 refund for the lamp, no refund for the mug, and a final message consistent with the recorded outcome. A different wording of the confirmation can still be correct.
Why the average score needs a closer look
Numan's engineering account, published by LangChain on 19 October 2026, reports that dataset averages concealed failures in particular categories. It also describes fixed conversation scripts becoming irrelevant when the agent answered differently from what the next scripted user message assumed. The team reports using category-specific scoring and running evaluations through the same agent graph used in production. Numan's evaluation account
The distinction between current and experimental work matters. Numan describes adaptive conversation simulation as a prototype that did not yet gate releases, pending clinician approval of its criteria. Automated online evaluation was a planned next step. That account supports an engineering discussion; it does not establish clinical safety. Prototype and future-work details
My takeaway for an ordinary support workflow is to decide which failures need their own release decision. In the refund example, better tone cannot compensate for refunding an item the customer decided to keep. Keep that failure visible even if it occurs in a small part of the test set.
Check the conversation, the actions and the records
LangChain's agent-evaluation FAQ distinguishes checks on individual steps, final responses, whole conversations and resulting system state. That gives a useful basis for examining the refund journey. Agent evaluation FAQ
Did the agent retain the current request?
Read the conversation with the customer's correction in view. Does the proposed refund contain only the lamp? Does the agent ask for confirmation of the revised £60 amount? Does it incorrectly treat the earlier request as permission to refund both items?
The test should allow a sensible clarification. If the customer says only “keep that one”, the assistant may need to establish which item they mean. An evaluator that demands immediate completion would reward guessing.
Did it take the permitted action?
Inspect the recorded tool calls and their results. In this scenario, no refund submission should occur before the revised confirmation. The submitted order, item, amount and payment destination must match the agreed request.
Check the sequence as well as the final record. Refunding both items and later correcting the balance still violates the fictional policy. A clean final state could conceal an action the customer never approved.
Do the records support the final message?
Inspect the refund records in the test system directly. Check the lamp's refund, the untouched mug and whether duplicate submissions created more than one refund. Then compare those records with what the assistant told the customer.
For a separate failure case, make the refund service return a definite rejection without creating a refund. The expected response should describe the failure and the agreed next step. Claiming that the refund succeeded should fail the check, however reassuring the wording sounds.
Keep uncertain outcomes separate. A timeout does not establish whether the refund happened. Define a recovery case in which the agent checks status before attempting another submission.
Make the test respond to the conversation
For the first test, I would have a teammate play the customer against an isolated test order. Give them the customer's goal and correction, and tell them when to introduce it. They should change the request when the assistant presents the initial refund for approval, then answer the assistant's actual questions.
This makes the intended branch explicit without requiring the agent to produce an exact sentence. If it submits the full refund before confirmation, record the failure immediately. Do not continue the script as though the earlier step worked.
For automation, the same scenario can use branching rules or a simulated user. Keep the simulator's instructions separate from the agent's instructions. Review whether it follows the customer's goal, introduces the correction at the intended point and stops at the agreed limit. A simulator that quietly accepts the wrong refund creates a misleading test.
Use the application's real orchestration and policy checks with isolated test services and accounts. Record any differences from production, particularly payment behaviour, permissions and retries. Nobody needs to move real customer money to exercise this scenario.
Anthropic's evaluation guidance recommends clean environments for separate trials and repeated trials because agent behaviour can vary. For this example, reset the order and refund records before each run, then retain every result rather than selecting the most convincing transcript. Anthropic's agent evaluation guidance
Agree who can approve the release
I would ask the support or product owner to approve the refund policy and examples, and engineering to demonstrate that the checks inspect the correct records. Name one person accountable for the release decision. In a small team, someone may hold several roles, but the responsibilities should still be explicit.
For this fictional workflow, I would treat a confirmed unauthorised refund, a duplicate refund or a false completion claim as a release blocker. Tone and brevity can be reviewed separately. Agree those decisions before seeing the new version's scores, so an attractive average does not change the standard halfway through review.
Use direct checks for facts such as amounts and item identifiers. A model judge can help assess whether a message explains the outcome clearly, but compare its decisions with human reviews. LangChain's FAQ recommends narrow judging questions and checking where model and human assessments disagree. Guidance on evaluation methods
The release record should identify the agent configuration, test cases, scoring criteria, failures and decision owner. Keep the comparison consistent: if you change the criteria, rerun the baseline as well. Otherwise, a score change may reflect a different test rather than a better agent.
If it is difficult to establish which service owns the refund or how retries affect it, that is an architecture review question to resolve alongside the evaluation work.
Frequently asked questions
Do single-turn tests still help?
Yes. Keep focused checks for behaviour one response can demonstrate. Add conversation tests for requirements that change or accumulate across turns. Passing one layer does not establish that the whole refund journey works.
How many scenarios should a team start with?
I would start with the normal refund journey and a few deliberate variations: a changed item, an ambiguous correction, a definite tool failure and an uncertain tool outcome. This is a starting set for discussion, not evidence of complete coverage. Expand it around the mistakes your product cannot afford and the failures you observe.
What should happen after release?
Assign someone to review failed and uncertain journeys, confirm what happened and turn useful cases into repeatable tests. LangChain distinguishes this production evaluation from the offline comparisons used before release. Offline and online evaluation
For teams planning an AI adoption pilot, I would write that review responsibility into the scope from the beginning. Before granting an agent the ability to change a customer record, be able to show what a correct change looks like and who will act when the evidence says it went wrong.