Task success and tool-call accuracy
For agents, the question is not whether the text reads well but whether the job got done and the right tools were called with the right arguments. A different measurement problem to text quality, and usually a more tractable one.
You order a takeaway. Forty minutes later a courier sends a cheerful message: "Delivered! Enjoy your meal."
There is no food at your door.
The message was perfect. The job was not done. And no amount of studying the wording of that message would have told you so — you had to go and look.
That is how you test an agent. Two questions, both about facts rather than words: did the thing happen, and did the steps make sense.
Two questions, not one
Did the parcel arrive? Go and look at the thing that was supposed to change. Is the refund in the payment system. Is the ticket closed. Does the row exist, with the right values. This is task success, and it is binary — no partial credit, no "mostly".
Did the courier drive sensibly? Read the steps. Right tool for the job, right arguments, no pointless detours, nothing dangerous. This is tool-call accuracy, and it is scored per step.
You need both because they fail independently. A run can reach the correct end state through a terrifying route that will break next week. A run can also do everything beautifully and still not finish the job.
The four words you will hear
Task success rate. Share of jobs where the real state ended up correct. The headline number.
Tool-call accuracy. Share of individual calls that were the right tool with the right arguments.
Step count. How many calls it took. Not a quality measure on its own, but a sudden jump means something changed.
Ground truth. What the world should look like afterwards. For an agent this is a database state, not a sentence — which makes it much easier to check than text.
Why you should care about this
Because "it worked when I tried it" is the most expensive sentence in agent testing.
Agents pick a fresh path every run. One good run tells you one good run happened. Without a success rate over several attempts, you are shipping on an anecdote — and the failures that matter are exactly the ones that only appear on the third attempt.
The second reason is that this is where you can be precise while everyone else is vague. A room full of people saying the agent "seems reliable" cannot argue with "18 of 20 jobs completed correctly; the two failures both refunded the wrong order, and here are the runs". That is a metric worth reporting.
The reply says "done". The database says otherwise. Only one of those two is a test result.
How you measure it
1. Write down the end state before you run anything. For each job: what should be true afterwards, expressed as something you can query. "Refund of 49.00 exists against order A-1001, ticket 4417 is closed, one email sent." This is your answer key, and it is far more objective than any text rubric.
2. Run each job at least three times. Report a rate, not a verdict. Three is a minimum; five is better for anything touching money.
3. Check the world, not the words. Query the database. Read the email catcher. Look at the payment sandbox. Only after that, read what the agent said — and note any gap between the two, because that gap is its own bug.
4. Score the calls separately. For each call: was this the right tool, were the arguments right, was it necessary. A run that succeeds with a wrong-but-harmless call is a warning about next month.
5. Count what should never happen. Calls to tools not needed for this job. Duplicate writes. Calls after the job was already complete. These are the early signs of the loop problems that come later on this layer.
6. Keep the runs. A trajectory you cannot re-read is a test you cannot explain. Save the tool calls with arguments and results — the same record-keeping that makes ordinary releases defensible.
Try this today
Pick one job your agent does. Write the end state as three queries. Run it three times, checking the world each time.
JOB: resolve ticket 4417 (customer charged twice for order A-1001)
END STATE I EXPECT
1 refund of 49.00 exists against order A-1001 (payment sandbox)
2 ticket 4417 status = closed (ticket API)
3 exactly one email to the customer (mail catcher)
RUN 1 refund: A-1001 ok ticket: closed emails: 1 PASS 6 calls
RUN 2 refund: A-1002 WRONG ticket: closed emails: 1 FAIL 6 calls
RUN 3 refund: A-1001 ok ticket: closed emails: 2 FAIL 9 calls
task success rate 1/3
wrong-argument calls 1 (refund on the wrong order)
duplicate writes 1 (second email after a retry)
WHAT THE AGENT SAID, ALL THREE TIMES
"I've refunded the duplicate charge and let the customer know."Three runs, three different outcomes, one identical closing message. That table is worth more than a page of prose about agent reliability, and you can produce it in an afternoon with no special tooling.
How to show you know it
A success rate with the failures attached. "1 of 3, and here is what went wrong in each." Rates without examples get argued with; rates with runs get fixed.
An end-state answer key. Three queries per job, written before the run. It shows you understand that for agents the oracle is a database, not a sentence.
The double-run bug. Almost every young agent has one. Finding it is cheap and it is the kind of bug people remember you for.
A step-level observation. "It calls get_customer twice on every run — harmless today, but it is the pattern that becomes a loop when the API is slow." Reading trajectories rather than answers is the skill this layer is built on.
Questions
What if I cannot see the database?
Then ask for read access to a test environment, and treat the request as a blocker rather than a nicety. Without a way to check the end state you can only review the agent's own account of its work, which is the thing you are supposed to be verifying.
The agent got there by a strange route but the result was right. Pass or fail?
Task success passes; tool-call accuracy fails. Report both. Strange routes are how you find out a tool description is misleading, and they get more expensive as the feature grows.
How many runs per job?
Three to see the variation, five for anything involving money or customer contact. If the results differ every time, that instability is itself the finding — the same reasoning as reporting a rate rather than a verdict.
Can a judge model score task success?
It can read the trajectory and guess. It cannot query your database, which is where the truth lives. Use code for the end-state checks and save the judge for the parts that need reading.