How to Test Features That Give a Different Answer Every Time

A tester on a support ticketing team spends Tuesday morning re running the same search query five times against a new AI summary feature. Each run returns a slightly different paragraph. Same ticket, same query, same button click, five different summaries. She flags it as a bug. The engineer closes it as expected behavior. Nobody on the team can agree on what "passing" even means anymore, and the sprint's test plan sits half finished because the old checklist assumes a fixed, checkable answer.

This is the new normal for any team shipping AI powered features, from smart search to auto generated summaries to recommendation widgets. The traditional test case was built on one quiet assumption: give the system the same input and it returns the same output, every time, forever. AI features break that assumption on purpose. This post is about what to do when they do, and how to keep writing test cases that mean something once "correct" stops being a single fixed answer.

Why the old test case model assumes a deterministic system

A classic test case reads like a small contract. Given this input, the system returns exactly this output. A login form with the right password returns a dashboard. A shopping cart with three items and a 10% coupon returns a specific subtotal. That contract works because the underlying logic is deterministic: the same code path runs the same way every time, so one expected result covers every future run.

Non-deterministic systems break that contract at the design level, not by accident. A large language model behind a summary feature samples from a probability distribution over possible next words. Run the same prompt twice and you can get two summaries that use different sentence structure, different word choice, and sometimes a different emphasis, while both are reasonable summaries of the same ticket. That is not a bug in the traditional sense. It is the system doing what it was built to do.

Teams that try to force this into the old model end up in one of two bad places. Either they write a test case with one exact expected string, which fails constantly and trains everyone to ignore red test runs, or they give up on documented cases for that feature entirely and test it by feel before each release. Neither approach scales past a two person team, and neither gives you anything to point to when a stakeholder asks whether the feature was actually checked before shipping.

Redesign the test case around acceptable ranges, not exact matches

The fix is not to abandon documentation. It is to change what the expected result column contains. Instead of one exact output, a test case for a non-deterministic feature documents the boundaries of an acceptable output, and the tester checks the actual result against those boundaries.

For a support ticket summarizer, a documented test case might specify:

• The summary must mention the customer's stated issue by name, not a generic paraphrase
• The summary must not exceed 3 sentences
• The summary must not invent a resolution the agent never offered
• The summary must not drop the ticket's stated urgency level if one was set

None of those four checks require the output to match a fixed string. All four are checkable by a human reading the result, and all four catch the failures that actually matter: a summary that hallucinates a refund that was never promised, or one that buries an urgent ticket's status.

This is the same discipline good testers already apply to date pickers and currency fields, just stretched further. A test case for select a date next month doesn't hardcode a specific calendar date either. It documents the rule (the picker advances one month and lands on the same day number) and the tester or script checks the rule holds. AI output test cases just need more rules per case, because the acceptable range is wider.

What to document when the output is a range, not a value

A test case for a non-deterministic feature should still have the same structural bones as any other case: a title, a precondition, steps, and an expected result. The expected result section is where the real change happens. Write it as a checklist of properties the output must have, not a single value it must equal.

Three properties are worth documenting on nearly every AI feature test case:

• Boundedness: does the output stay inside a stated length, tone, or format limit
• Groundedness: does the output only state things present in the source input, with no invented facts
• Consistency of intent: across repeated runs, does the output serve the same purpose even when the wording changes

A team testing an AI generated product description feature might add a fourth property specific to that feature, like never states a price, since price is pulled from a separate system. The point isn't a universal template. It's that each property is something a tester can look at an actual output and mark pass or fail on, the same muscle memory as any other manual check.

::: callout tone=tip Write the must not list before the must list. It's usually easier to name three things an AI output should never do (invent a fact, drop a safety warning, exceed a length limit) than to name the one thing it should always say. :::

Where repeated runs actually earn their keep

Because a single run of a non-deterministic feature proves less than a single run of a deterministic one, repetition becomes part of the method, not a wasted retest. Running the same test case three or five times against the same input, with the same property checklist applied each time, tells you something a single pass never could: whether the failure was one bad sample or a pattern.

Consider a real week for a QA lead at a mid sized SaaS company: the team ships an AI generated changelog summary feature on a Wednesday, and a tester runs the same release notes through it four times before sign off. Three runs pass every property. One run invents a feature that wasn't actually in the release. That's not a stable feature yet, and now there's a documented run log showing exactly why the release got held rather than someone's vague sense that it felt off.

This is where a documented test case earns its cost back. If the only record is I ran it a few times and it seemed fine, nobody downstream, not the next tester, not the engineer fixing it, not the person writing the release notes, can tell what was actually checked. A logged run against a written property checklist gives all three of them the same evidence.

Keeping the rest of your suite honest

None of this means every test case in the suite needs a fuzzy expected result. Most of a typical application, the login flow, the checkout math, the permissions model, is still deterministic and should still be tested with exact expected values. The mistake teams make is letting the AI feature's uncertainty bleed into how they write cases for everything else, loosening standards suite wide because one feature is genuinely unpredictable.

Keep the two styles of test case clearly separated in your test management practice. A tester scanning the suite should be able to tell at a glance which cases expect an exact match and which expect a property check, so nobody mistakes a legitimately variable AI result for a broken deterministic one, or vice versa.

Where this leaves teams shipping AI features next quarter

Non-deterministic output isn't a temporary annoyance that better models will eventually fix. It's a structural property of how these features work, and it's likely to show up in more of the product surface over time, not less. Teams that redesign their test case format now, around properties instead of exact values, end up with documentation that holds up as more of the product goes this direction. Teams that keep forcing exact match cases onto probabilistic features end up with a suite everyone quietly ignores.

The tester from Tuesday morning didn't need a different tool to solve her problem. She needed a different kind of expected result column, one that asked does this output stay inside the lines instead of does this output match exactly.