Test oracles: how you know it’s right

A test oracle is the source of truth you compare an output against, so you can say a result is correct rather than merely plausible. Every test has one, even when nobody has named it.

The oracle problem is the name for what happens when no such source exists. You can see the output and you have no dependable way to decide whether it is right. That used to be a niche academic phrase, and AI features have made it an everyday one.

Here is what it looks like when a team has not noticed the gap.

A junior tester once asked a question that stopped a room.

The team was testing a currency converter. Somebody typed 100 dollars, the screen said 78 pounds, and everybody nodded. Then she asked: "How do we know 78 is right?"

Nobody could answer. The number looked plausible. It came out of the system they were testing, so agreeing with it proved nothing. Somebody eventually opened a bank website, and the real answer was 74.

The thing they were missing has a name. It is called a test oracle, and it is the source of truth you compare an answer against.

The terms you will hear

• Test oracle. Whatever tells you the output was correct: a spec, a value, a rule, a person.
• Expected result. The specific value you compare against. The oracle is where that value came from.
• Oracle problem. Having no reliable source of truth for the output you are looking at.
• Partial oracle. A source that can rule some answers out without confirming the right one, such as "the total must equal the sum of the lines".
• Consistency oracle. Comparing against a previous run or another implementation. Useful for spotting change, and it proves nothing about correctness on its own.

Five oracles you already use

You use these daily without naming them, and naming them makes gaps visible.

• A stated expected value. The acceptance criteria say refunds take 14 days, so 14 is correct. Turning a story into cases is where that value comes from.
• A known-good source. A bank rate, a tax table, the previous release's output, a supplier's price list.
• A second implementation. Work the sum out yourself, in a spreadsheet or a script, and compare.
• A rule that must hold. The parts must sum to the total. A refund can never exceed what was paid. Nobody sees another customer's data.
• A person who knows. Sometimes the only oracle is the finance lead, and asking is faster than deducing.

Why this matters more every year

For most of software history, oracles were easy. A spec said what to expect and you compared. That is why the idea sounds academic.

Then features started producing answers nobody can look up. A recommendation. A summary. A support reply written by a model. Ask what the correct output is and there is no single right answer, which means there is no value to compare against.

That is the whole reason assertions break on AI features. The old oracle disappeared, and the replacement is a set of properties that must hold plus a scored judgement. The idea in this topic is the foundation the entire AI testing layer sits on.

How to use it deliberately

1. Before writing a test, say where the expected result comes from. Out loud. If the answer is "it looks right", stop and go and find a real source.
2. Never let the system be its own oracle. Refreshing a snapshot until the test passes is the same error in a nicer wrapper.
3. Prefer a rule when a value is unavailable. For example: the sum of the line items plus delivery must equal the total charged, whatever those numbers happen to be.
4. Write the oracle into the test case. One line saying where the expectation came from turns future arguments into a lookup. It belongs next to the expected result in a case somebody else can run.
5. Ask who owns the answer. For money, tax, medical or legal outputs, the oracle is a person or a document, and you want their name on it.
6. Where nothing exists, say so. "There is no source of truth for this output" is a genuine finding, and it usually reveals a requirement nobody wrote.

A worked example

Here is the same currency screen tested with the oracle named in every row. It takes a few extra seconds per case and it changes what the suite proves.

The last block is the point. Naming the oracle found two missing requirements, and no amount of clicking would have surfaced them. That is testing before the test.

How to show you know it

• An oracle column in your cases. Rare, cheap, and it makes reviewers trust the suite.
• A missing oracle you turned into a requirement. The strongest version of this skill. "Nobody had decided what a stale rate should do."
• A property test where no value existed. Showing you can assert that parts sum to a total, when the total itself varies, is exactly what AI features need.
• A refused snapshot. "I did not update the stored output to match, because that would make the system its own oracle." Say that once and people remember it.