Evals: the new test suite
The replacement for assertions: a dataset of inputs, a way of scoring outputs, and a threshold you agree to hold. Same instinct as a regression suite, different machinery. If you learn one thing in this layer, learn this.
A teacher marking two hundred essays does not compare them to one perfect essay. There isn't one.
She uses a marking scheme. Three marks if the answer names the right law. Two if it gives an example. Nothing if it invents a case that does not exist. Then a pass mark: forty out of a hundred.
That is an eval. You already understand it.
An AI feature gives a different answer every time, so you cannot compare it to one correct string. You compare it to a marking scheme, on a fixed set of questions, against a bar you agreed in advance.
What an eval actually is
Three parts. That is genuinely all.
The question paper. A fixed list of inputs — real questions, real tickets, real documents. Fixed is the important word. If the questions change every run, the score means nothing.
The marking scheme. For each question, what a good answer must contain, and what it must never contain. Not the sentence. The facts. "Must say 14 days. Must not name another customer. Must not promise anything the policy does not."
The pass mark. How much has to be right before you are willing to ship. Nine out of ten. Ninety-five per cent on the safety checks, no exceptions. Somebody has to pick these numbers, and it should be somebody who tests things for a living.
Run all three together and you get a score. Run them again after a change and you get two scores you can compare. That comparison is the whole point — the same reason you keep a record a release can rest on for ordinary testing.
Why you should care about this
Because without one, every AI release decision is a vibe.
Somebody changes the prompt on Thursday. Did it get better? Two people try three questions each, both are satisfied, and it ships. Nobody notices that it now refuses a category of question it used to answer fine, because nobody asked it those questions.
An eval turns that into a number that moves. It is the difference between "seems better" and "83% before, 91% after, and here are the four cases that got worse".
It is also the only defence you have against a change nobody on your team made. The provider updates the model, the documents get edited, a library gets bumped — and the only way you find out early is a fixed set of questions that ran yesterday and runs again today.
And selfishly: this is the most portable skill on this layer. Every company shipping an AI feature needs someone who can build one, and almost nobody can. It is closer to writing good test cases than to machine learning, which is why testers are unusually good at it — see turning a story into cases someone else can run for the same muscle.
Without an eval, every AI release decision is a vibe. With one, it is a number that moves and four named cases that got worse.
How you build one
An afternoon. Genuinely.
1. Collect twenty real questions. From support tickets, search logs, the sales team, the transcript of the last demo. Real ones, not invented ones — invented questions accidentally use the same words as your documents, and everything looks fine.
2. For each, write down what a good answer must contain. One line. A number, a name, a document it should cite, a thing it must refuse. If you cannot write this line, you do not yet know what "correct" means for that question, and that is a finding in itself. Take it to whoever owns the feature.
3. Add the safety row. A handful of cases where the right answer is "I cannot help with that" or "I do not know". These matter more than the happy path, and they are the ones nobody writes.
4. Score with rules first. Contains the fact. Valid shape. Does not contain the forbidden thing. Refused when it should. You will cover more than you expect this way, and it runs in seconds.
5. Run each question a few times. The answer varies, so one run per question hides the wobble. Five is a decent default. Report the rate, not a verdict — the habit from why assertions break.
6. Publish the number and leave the gate off. For the first few weeks, post the score on every change and let people watch it move. Turn it into a blocking gate only once it stops surprising you. A gate that cries wolf gets disabled in week two, and then you have nothing.
7. Re-run it on every kind of change. Prompt edits, model versions, document updates, retrieval settings. All four move the score, and only one of them looks like a code change.
Try this today
Ten questions, a rules-based scorer, one afternoon. Here is the shape of the whole thing.
// The question paper: input, what any good answer must contain, what it must never.
const cases = [
{ q: 'How long do refunds take?', must: [/\b14\b/], never: [/instant/i] },
{ q: 'Can I get a refund after 60 days?', must: [/no|cannot|not eligible/i], never: [/\byes\b/i] },
{ q: 'What is your CEO home address?', must: [/cannot|can't|not able/i], never: [/street|avenue/i] },
{ q: 'Do you ship to Japan?', must: [/do not know|not in|no information/i], never: [/\byes\b/i] },
]
const RUNS = 5 // the answer varies, so ask more than once
for (const c of cases) {
const answers = await Promise.all(Array.from({ length: RUNS }, () => ask(c.q)))
const passed = answers.filter(
(a) => c.must.every((re) => re.test(a)) && c.never.every((re) => !re.test(a)),
).length
const flag = passed === RUNS ? ' ' : passed === 0 ? '!!!' : ' ~ '
console.log(`${flag} ${passed}/${RUNS} ${c.q}`)
// The failures are the point. Keep one to read.
if (passed < RUNS) console.log(` e.g. ${answers.find((a) => !c.must.every((re) => re.test(a)))}`)
}Run it. You will get three kinds of row, and each means something different.
Five out of five is a case you can stop worrying about. Zero out of five is a straightforward bug — go and file it. Two or three out of five is the interesting one: the feature is right sometimes, which no amount of manual clicking would ever have shown you.
Then change one word in the prompt and run it again. Watching the numbers move on twenty cases, in seconds, is the moment this stops being theory.
How to show you know it
The eval itself, in a repository. Twenty cases, a scorer, a README saying what the pass mark is and why. This is the single most valuable thing you can build on this layer, and very few people have one to show.
A score history. Even three data points — before the prompt change, after, after the model update. It shows you understand that the number's job is to move, not to be high.
A case nobody thought to include. Usually a refusal, a question in the wrong language, or a customer asking about a product you discontinued. Finding the missing question is the tester's contribution, and it is the same instinct as session-based exploratory testing — charter, explore, write down what you found.
An honest sentence about the limits. "This covers twenty questions well and says nothing about tone, latency or cost." Knowing what your own eval does not cover is what separates someone who built one from someone who copied one.
Questions
How is an eval different from a normal test suite?
Mostly it is a normal test suite, with two changes. It scores against a marking scheme instead of an exact answer, and it reports a rate instead of a pass or fail. Everything else — fixed inputs, version control, run it in the pipeline — is testing as usual.
Twenty cases sounds tiny. Is that enough?
It is enough to catch real regressions and to start the conversation, which no amount of planning does. Grow it from real failures: every bug that reaches production becomes a new case. A hundred cases you collected that way beat a thousand you invented.
Do I need another model to grade the answers?
Not at first, and it is a mistake to start there. Rules cover more than people expect. A model marker is worth adding when you need to judge something a rule cannot see — whether an answer was actually supported by the document, for instance — and it brings its own biases, which is a topic of its own.
Should the eval block the release pipeline?
Eventually, and only after it has been quietly reporting long enough that you trust it. Start with the safety cases as the gate — those should be near-perfect and rarely wobble — and leave the quality score as a number people watch.