Smoke vs Sanity vs Regression Testing: Three Words, One Confusion

Search for smoke testing vs sanity testing and you will find a table of definitions that does not survive contact with a real Thursday afternoon. In most standups the three words are used interchangeably. Somebody says they will smoke test the fix, runs a handful of checks around the change, and everyone nods. What they actually did was a sanity check.

Nobody minds, and nothing goes wrong that day. It goes wrong later, in two specific places. A release document describes three steps that nobody performs separately. Or a new joiner is told to run a regression check before lunch and comes back four hours later. This post separates the three on the five things that actually differ, then runs one real change through all of them.

The short answer

Each of the three answers a different question.

• Smoke testing asks: is this build worth testing at all?
• Sanity testing asks: did this specific change work, and did it break anything next to it?
• Regression testing asks: does everything that used to work still work?

If you remember nothing else, remember the scope. Smoke is scoped to a build, sanity to a change, regression to the product. Everything below follows from that.

The five differences that matter

Scope

• Smoke: a fixed set of cases covering the critical paths, the same every time
• Sanity: cases chosen after you see what the diff touched, different every time
• Regression: every case in the suite that still earns its runtime

What triggers it

• Smoke: every build, automatically, with no human deciding
• Sanity: a specific fix or small change, usually when there is no time for the full suite
• Regression: a release, a sprint boundary, or a nightly schedule

Who owns a failure

• Smoke: the author of the last merged change, by default, until proven otherwise
• Sanity: whoever made the change, because judging blast radius needs knowledge of the diff
• Regression: the QA owner for that area, because a failure may be a real defect or a stale case

How long it may take

• Smoke: under ten minutes, or people quietly stop running it on every build
• Sanity: five to twenty minutes, because its purpose is fitting in the gap before a release
• Regression: as long as it takes, which is why it cannot be the gate for either of the others

What a failure blocks

• Smoke: all testing. The build is rejected and nothing downstream starts.
• Sanity: that change only. Other work continues.
• Regression: the release, subject to triage on severity.

That last set is the one worth internalising. The three suites are not ranked by importance. They are ranked by how much they stop when they go red.

One change, run through all three

A customer reports that business accounts are charged 1.9 percent where the pricing page promises 1.4 percent. A developer fixes the rounding helper and pushes at half past four.

The smoke run happens automatically when the build deploys. It runs the same twelve cases it always runs:

• The app loads and the sign in screen appears
• A known good user signs in
• The dashboard shows an account balance
• A payment submits and returns a reference
• The daily cap blocks a payment above 10,000.00
• The health endpoint reports every dependency up

Six minutes. It says nothing whatsoever about the fee bug, and it is not supposed to.

The sanity check is chosen after looking at the diff. Five cases, about four minutes:

• The business tier now shows 1.4 percent
• The enterprise tier at 1.1 percent is unchanged
• The personal tier at 2.4 percent is unchanged
• A 500.00 GBP payment debits 507.00, matching what the screen showed
• The daily cap still blocks a payment above 10,000.00

The fifth is there only because the cap and the fee share the rounding helper. No fixed suite would have selected it.

The regression run happens that night. All 840 cases, just over two hours. It is the run that catches a break in the refund calculation, three screens away. Nobody thought to check refunds at half past four, because nobody knew that screen read the same helper.

All three were necessary. None substitutes for another. The fix shipped on the sanity check and the regression suite confirmed it overnight, which is the normal and correct sequence.

Where teams get this wrong

Three failure patterns cover most of it.

One suite doing all three jobs. A single set of cases runs on every build, after every fix, and before every release. It is too slow to be a smoke gate and too narrow to be a regression suite. The symptom is a suite that takes 25 minutes and gets skipped under pressure.

A regression suite used as a gate. Somebody wires the full suite into the merge check. It takes two hours, developers start merging around it, and within a month the check is advisory. If you want a gate, it has to be a smoke gate.

A smoke suite that has grown. Cases get added after every production incident and nothing is ever removed. This is the most common of the three and the hardest to see, because each individual addition was justified.

The fix for the third is to hold runtime as a fixed constraint. A new case then has to displace an existing one rather than simply join the queue.

That is really a pruning problem, and pruning never gets scheduled. Deciding what stays is the same judgement as deciding what to automate in the first place. The same reasoning covers the cases that should not be automated at all.

The vocabulary question

A fair objection: does any of this matter if the team understands each other?

Mostly not. Say your team runs one short check after a deploy and calls it a smoke test. If everybody knows what that means, the terminology is doing its job. Precision you do not use is overhead.

It starts to matter at three points:

• When the process is written down for someone outside the team, such as an auditor or an enterprise customer
• When somebody new joins and has to infer the process from the words
• When the suites are automated, because a pipeline needs to know which set runs when

At that point the words become configuration rather than conversation. The differences above are then worth getting right. The wider map of testing types covers where the rest of the vocabulary fits.

Whichever names you use, write down which cases belong to which suite. That record is ordinary test case management work. It is what stops a smoke suite quietly growing into a regression run over two years.