Smoke Testing: The Check That Decides Whether Testing Starts

A build lands at nine in the morning. Somebody picks it up and works through the regression suite for four hours. At ten past three they discover that login has been broken since the first commit of the day. Every result from that morning is now worthless, because none of it ran against a working product.

That afternoon cost three testers most of a day. The check that would have saved it takes about six minutes. Smoke testing is that check. This post covers what belongs in the suite, how long it is allowed to take, and what the cases look like once you write them down.

What smoke testing actually is

Smoke testing is a short set of checks that answers one question. Is this build worth testing at all?

That is the whole purpose. A smoke suite is not trying to find bugs. It is trying to find out whether the deeper testing you are about to do will mean anything. If the application will not start, or the main screen will not load, or nobody can sign in, then every other result you collect that day is noise.

The name comes from hardware. You power the board up and see whether smoke comes out. Nobody running that check believes it proves the board works. It proves the board is not obviously dead, which is a different and much cheaper claim.

Most teams already do a version of this. Somebody opens the app after a deploy, clicks two or three things, and says it looks fine. That instinct is right. The gap is that the check lives in one person's head. It varies by who ran it, and nobody can say afterwards what was covered.

The gate, and what it costs to hold it open

The useful way to think about a smoke suite is as a gate with a price attached.

Holding the gate closed costs you the runtime of the suite. Every build pays it. Opening the gate on a broken build costs you whatever testing happens next, which is measured in hours or days rather than minutes.

That asymmetry is why smoke suites are worth running even when they almost never fail. Say the suite catches one dead build a month on a team of four testers. It has already paid for a year of its own runtime.

It also explains the constraint that matters most. If the gate is expensive to hold closed, people stop closing it.

Ten minutes is not a law. It is the point where a developer waiting on a result starts doing something else, and the feedback loop breaks. Some teams hold the line at five. Very few hold it past fifteen.

How many cases belong in it

Somewhere between eight and twenty, for most products.

The number matters less than the rule you use to pick them. A case earns a place in the smoke suite if its failure would make the rest of the day's testing meaningless. That is a much narrower test than "this feature is important".

Applying it honestly removes most candidates:

• Password reset is important, but if it is broken the rest of the suite still runs. Not smoke.
• Sign in is not more important than password reset. If it is broken, though, nothing else can be reached. Smoke.
• The reporting export is used by every customer, and its failure blocks nothing upstream. Not smoke.
• The payment submission endpoint returning a 500 makes half the suite unrunnable. Smoke.

Teams tend to grow smoke suites by addition. Somebody gets burned by a bug that reached production, and a case is added so it never happens again. Two years later the suite takes twenty six minutes and runs twice a week.

The fix is to hold runtime as the constraint. Every new case then has to compete for a slot rather than simply join the queue.

When the smoke suite itself starts flaking

This is the failure nobody plans for, and it is worse than having no smoke suite at all.

A gate that fails at random teaches people to ignore it. The first time it goes red on a good build, somebody re-runs it. The second time, somebody re-runs it without looking. By the fourth time the team has learned that red means nothing. The gate now costs runtime while providing no signal.

At that point a genuinely broken build sails straight through. The one check that would have caught it has already been classified as noise.

Smoke cases are unusually exposed to this. They run against a freshly deployed environment, often before caches are warm and background jobs have settled. A case that passes locally will fail in the first thirty seconds after a deploy, for reasons that have nothing to do with the build.

Two things help. The first is a readiness check before the suite starts, so the gate does not open until the environment reports itself up.

The second is treating a flaky smoke case as a priority defect rather than an annoyance. Flaky tests are a design problem rather than bad luck, and the smoke suite is the worst place in the codebase to tolerate one. If you are not sure which of your cases are unreliable, your run history will tell you before your team does.

What the cases look like written down

Here is a smoke suite for a payments application. The product takes a payment, applies a fee that varies by tier, enforces a minimum transaction value, and caps how much a user can send in a day.

Twelve cases, in the order they run:

• The application loads and returns the sign in screen
• A known good user signs in with valid credentials
• The dashboard renders and shows an account balance
• The payment form opens from the dashboard
• A payment at the minimum allowed value is accepted
• A payment one penny below the minimum is refused, with the correct message
• The fee shown on screen matches the tier for the amount entered
• A submitted payment returns a confirmation reference
• The new payment appears in the transaction list
• A payment that exceeds the daily cap is blocked
• Sign out ends the session and returns to the sign in screen
• The health endpoint reports every dependency as available

That list takes roughly six minutes to run. It touches authentication, the main read path, the main write path, the two business rules that define the product, and the state of the dependencies.

Written as a documented case, the sixth one looks like this.

Title: Payment below the minimum value is refused

Preconditions: A known good user is signed in. The account balance is above 100.00 GBP. The configured minimum transaction value is 1.00 GBP.

Steps:

1. Open the payment form from the dashboard
2. Enter a valid recipient
3. Enter 0.99 in the amount field
4. Submit the payment

Expected result: The payment is refused. The form stays open with the entered values intact. An error appears against the amount field reading "The minimum payment is 1.00 GBP." No transaction is created, and the account balance is unchanged.

That last sentence is the part most smoke cases leave out. "It shows an error" passes on a build that shows an error and takes the money anyway. Naming what must not have happened is what turns a click-through into a check.

Where the suite runs, and who owns a red result

A smoke suite usually runs in three places, and the three are easy to confuse.

• On a pull request, against a preview environment, to tell the author whether the branch is viable
• On merge to the main branch, to protect everyone else from a broken trunk
• After a deploy to a shared test environment, to tell the QA team whether the day can start

The third is the one this post is about, and it is the one most often missing. Teams that have solid pipeline checks still walk into the ten past three problem, because nothing verified the environment testers actually use.

Ownership is the other half. A red smoke result has to belong to somebody by default, before anyone argues about it. The usual rule that works is that the author of the last merged change owns it until proven otherwise. That is not a judgement about blame. It is a way of making sure the result gets looked at within minutes rather than sitting in a channel.

Write the rule down next to the suite. A gate with no named owner is a gate that stays red for an afternoon while three people each assume somebody else is on it.

What deliberately stays out

A smoke suite is defined as much by its exclusions. These do not belong in it, however important they are:

• Boundary and edge case coverage, which belongs in the functional suite
• Anything needing test data that has to be built during the run
• Cases that depend on a third party sandbox being available
• Anything that takes more than about thirty seconds on its own
• Performance assertions, because a cold environment fails them for the wrong reason

The last one catches people out. A smoke case asserting that the dashboard loads in under two seconds will fail on the first request after a deploy. It will be re-run rather than investigated, and you are back to a gate nobody believes.

Where the smoke suite sits

Smoke testing is one gate in a longer sequence, and it is the cheapest one. It runs on every build. Sanity checks run after a targeted fix. The regression suite runs before a release.

Each answers a different question. Confusing them is why so many teams have one suite doing all three jobs badly. The wider map of testing types sets out how the layers fit together.

The practical advice is narrow. Write the smoke cases down, keep them under ten minutes, and hold the runtime as a hard constraint rather than a target. When somebody wants to add a case, ask which one comes out. If the answer is none, the suite is on its way to becoming something else.

Keeping that record current is ordinary test case management work. The smoke suite is small enough to be easy to maintain, and important enough to be expensive to neglect. That is an unusual and useful combination.

If you are deciding which of the twelve to automate first, the scoring method for automation candidates applies here more cleanly than anywhere else in the suite.