Agent trajectory testing
Judging the path, not just the destination. An agent that reaches the right answer after fourteen wasted calls and one dangerous one has not passed. Learning to assert over a sequence of steps is the core new automation skill here.
In a maths exam you get marks for the working.
Write the right answer with no working and you lose most of them. Write good working and slip on the final line and you keep most of them. The examiner cares how you got there, because the working is what tells them whether you can do it again next week.
An agent's working is its list of tool calls. Testing that list, rather than the final message, is the main new automation skill on this layer.
What a trajectory is
One job, every step, in order, with arguments and results.
```
get_ticket(4417) → get_customer(8812) → search_docs("refund policy")
→ get_orders(8812) → refund(A-1001, 49.00) → send_email(...) → close_ticket(4417)
```
Seven steps. Read it like an examiner reading working. Did it check the policy before refunding, or after? Did it look up which order was the duplicate, or guess? Is there a step here that should never happen without a human saying yes?
The working is what tells you whether it can do it again next week. The final answer only tells you about today.
The five rules worth asserting
Forbidden call. delete_customer must never appear in a refund job. One line, enormous value.
Order. refund must be preceded by get_orders and by a policy lookup. Acting before checking is the most common dangerous pattern.
At most once. send_email no more than once per job. refund exactly once. This catches the retry-that-duplicates bug.
Required check. For anything irreversible, a verification step must appear before it. If your design says a human approves refunds over 100, the approval call must be in the trajectory.
Termination. The run ends within a step budget, and ends deliberately rather than by hitting the ceiling.
Why you should care about this
Because it is the only way to test an agent that is right for the wrong reasons.
A run can reach the correct end state having called a tool it should not have been able to reach, or having skipped the policy check because the answer happened to be in its context already. Both pass an outcome test. Both are next month's incident.
It is also the part of this layer that looks most like the testing you already do. These are invariants — the same shape as "the audit row must exist" or "the email must be sent exactly once" — and writing them well has more to do with knowing the domain than knowing anything about models. It pairs directly with task success: one checks the destination, this one checks the journey.
How you test it
1. Get the trajectory as data. JSON, one entry per call, with tool name, arguments, result and timestamp. If all you have is a prose transcript, ask for structured logs — this is the enabling request for everything else here.
2. Write the rules with the person who owns the feature. Ten minutes and a whiteboard. "Never delete. Refund only after checking the order. One email. Approval above a hundred." You will find at least one rule nobody had decided, which is worth the meeting on its own.
3. Turn each rule into a check over the list. Plain code — filters and index comparisons. No model needed, and it runs in milliseconds.
4. Run each job several times and check every run. Variation is normal; a rule violation on run three is a real bug even if runs one and two were clean.
5. Diff trajectories across versions. When the prompt or the model changes, compare step counts and shapes. A job that used to take six calls and now takes eleven is telling you something before any quality score moves.
6. Keep the interesting ones as fixtures. A trajectory that broke a rule is a regression test forever — you can replay the rules against the stored run without calling the model at all.
Right answer, dangerous route, is not a pass. It is a bug you have agreed to be surprised by later.
Try this today
Take one job, get its trajectory as JSON, and write four rules.
type Call = { tool: string; args: Record<string, unknown> }
const rules = [
{
name: 'never deletes anything',
check: (t: Call[]) => !t.some((c) => c.tool.startsWith('delete_')),
},
{
name: 'checks the orders before refunding',
check: (t: Call[]) => {
const refund = t.findIndex((c) => c.tool === 'refund')
const orders = t.findIndex((c) => c.tool === 'get_orders')
return refund === -1 || (orders !== -1 && orders < refund)
},
},
{
name: 'emails the customer at most once',
check: (t: Call[]) => t.filter((c) => c.tool === 'send_email').length <= 1,
},
{
name: 'finishes within 15 steps',
check: (t: Call[]) => t.length <= 15,
},
]
for (const run of runs) {
const broken = rules.filter((r) => !r.check(run.calls)).map((r) => r.name)
console.log(`${run.id} ${run.calls.length} calls ${broken.length ? 'FAIL: ' + broken.join('; ') : 'ok'}`)
}Four rules, twenty lines, and it runs against every stored trajectory you have. Point it at last week's runs and you will usually find a violation nobody noticed, because nothing was watching the working — only the answer.
How to show you know it
A rules file with the domain in it. Not generic advice — your product's actual constraints, written as checks. This is the artefact that shows you understand both testing and the feature.
A run that passed the outcome and broke a rule. The single most convincing example on this layer, and the reason trajectory testing exists.
A step-count diff across versions. "Six calls before the model update, eleven after, same success rate." Early warning, cheaply produced.
A rule nobody had decided. "Should it be able to email a customer without approval? Nobody knew, so we decided." That is the work — the same value as being the quality voice in a design review, one layer up.
Questions
Is this not just integration testing?
It is integration testing where the caller decides what to call. The invariants are familiar; what is new is that the sequence is generated fresh each run, so you assert rules rather than a script.
How many rules do I need?
Start with three: never do the irreversible thing, check before acting, and terminate. Those three cover most of the risk. Grow the list from real runs rather than from imagination.
What if the agent is right but takes twenty steps?
Record it and set a budget. Long runs cost money and latency, and a step count that climbs over releases is a reliable early signal that something upstream changed.
Do I need the model to judge the trajectory?
Rarely. Most useful rules are structural — which tool, in what order, how many times — and plain code checks them exactly. Reach for a judge only for questions like "was this detour reasonable", and treat its answer as a hint.