Why assertions break: non-determinism and drift
The same input gives a different output, and the model changes underneath you. `expect(response).toBe(...)` is simply not available. Understanding precisely why is what makes the rest of this layer make sense.
Ask five friends to describe the same photo in one sentence.
You get five different sentences. Every one of them is correct. Not one of them matches another word for word.
That is an AI feature. And it is why the first test you write fails.
expect(answer).toBe('Refunds take 14 days.')The answer this time was "You'll get your money back within 14 days." Correct. Helpful. Red.
Two reasons the answer moves
Reason one: the wording is free. There are a hundred correct ways to say the same thing, and the model picks one. Nothing is broken. This is like your five friends and the photo.
Reason two: the recipe changes. The provider ships a new version of the model. Nobody told you. Yesterday's careful wording is now slightly different — usually better, occasionally worse, always different. This is the café changing its coffee supplier: same menu, same price, drink tastes new.
Old tests assume both of these are impossible. That is why they break, and why teams end up deleting them and testing nothing at all — which is the worst outcome available.
Why you should care about this
Because your instinct is right, and only your method is wrong.
You do not actually care that the answer says "14 days" in exactly those words. You care that a customer is told the correct number, that nothing invented crept in, that the app does not promise a refund it cannot give, and that it says "I do not know" instead of guessing.
Those are all testable. They are just not testable with toBe.
The other reason is practical. When a feature can pass and fail on the same input, "it works" becomes a probability rather than a fact — and someone has to decide what probability is good enough to ship. If a tester does not put a number on it, the loudest person in the room will pick one by feel. That is exactly the ground QA metrics worth reporting covers.
How you test it
Stop asking "is this the right sentence?" Start asking "what must be true of any right sentence?" Then assert that.
Four kinds, in the order I would add them.
1. The fact is present. The number, the date, the name, the ID. Any correct answer to "how long do refunds take" contains 14. Search for the fact, not the sentence. This alone catches most real regressions.
2. The shape is right. If the app depends on JSON, valid JSON with the required fields is not a matter of opinion. Shape assertions never flake, so put them in first and let them run on every commit.
3. The forbidden thing is absent. No other customer's name. No internal note. No promise of a refund when the policy says none. No made-up product. This is your safety net, and it stays valuable forever.
4. It refuses when it should. Ask something out of scope, or something the documents cannot answer. The answer must be a clean "I do not know". Do not check the exact apology — check that nothing was invented.
Then two habits around all four.
Run it more than once. Five or ten times per case, and record how often it holds. You are no longer collecting pass or fail; you are collecting a rate. "Holds 10 of 10" and "holds 6 of 10" are different products, and the second one is a bug even though it passed the first time you tried.
Pin the version, then re-run on purpose. Ask which exact model version is in use and get it written down. When it changes, run the whole set again and compare the rates before and after.
That is how you catch the café's new coffee supplier on the day it happens, instead of three weeks later through a customer complaint. Some of this will feel like chasing flaky tests. The difference is that here the variation is the product working as designed, so quarantining it is not an option.
You do not care that the answer says "14 days" in those exact words. You care that the customer is told 14. Test the second thing.
Try this today
Fifteen minutes and one question.
Pick a question your AI feature should answer. Run it ten times. Paste the ten answers into a file side by side.
Read down the column and mark what is identical in every one, and what changes. The identical parts are your assertions. The changing parts are the wording you must stop testing.
// Ten runs of one question, then three assertions that do not care about wording.
const answers: string[] = await Promise.all(
Array.from({ length: 10 }, () => ask('How long do refunds take?')),
)
const checks = {
'states 14 days': (a: string) => /\b14\b/.test(a),
'no other customer named': (a: string) => !/\b(Priya|Daniel|Aisha)\b/.test(a),
'no invented promise': (a: string) => !/guarantee|instantly|immediately/i.test(a),
}
for (const [name, holds] of Object.entries(checks)) {
const passed = answers.filter(holds).length
console.log(`${passed}/10 ${name}`)
}
// Unique wordings, just to see the spread with your own eyes.
console.log(`${new Set(answers.map((a) => a.trim())).size} different wordings out of 10`)The last line is the one that changes how people think. Ten runs often produce nine or ten different wordings — and all three checks still pass ten out of ten. That gap is the whole lesson: the words are noise, the facts are the product.
Keep the file. When the model version changes, run it again and compare. You have just built the smallest useful version of an eval suite, which is the next topic on this layer.
How to show you know it
A before-and-after table. The brittle assertion, the property assertion that replaced it, and the pass rate over ten runs. This is the single most persuasive artefact on this layer, because everyone in the room has been bitten by a flaky AI test.
A stability number in your bug reports. "Fails 4 times in 10" is a completely different bug from "fails sometimes". One gets prioritised; the other gets closed as cannot-reproduce, which is the same lesson as writing a bug report that gets fixed applied to a moving target.
One question in the next planning meeting. "Which model version are we on, and who tells us when it changes?" If nobody knows, you have found a real gap in the release process, and it costs nothing to ask. It is the same move as deciding what to automate first — the cheap question that reorders the work.
Questions
Can I make it deterministic and go back to exact matching?
You can reduce the variation with settings, and some teams do for narrow tasks. You cannot rely on it — a provider update moves the output regardless, and a feature that is only correct in one exact wording is fragile anyway. Assert properties and the problem stops mattering.
How many runs is enough?
Five while you are exploring, ten in a suite you trust, more for anything involving money or safety. What matters is that you report a rate rather than a verdict.
Is this what people mean by flaky tests?
Not quite. A flaky test is a test that lies about a stable product. Here the product genuinely varies, and your test has to describe that honestly. The fix for a flaky test is to remove the randomness; the fix here is to assert the parts that do not vary.
My team wants a single pass or fail in the pipeline. What do I give them?
A threshold. "This case must hold in at least 9 of 10 runs" is a pass or fail, and it is honest. Agreeing the number is the interesting conversation, and it is the whole subject of evals.