Flakiness: causes, quarantine, retry policy

A flaky test passes and fails on the same code, with no change in between.

The important word is same. A test that fails because the product is broken is doing its job. A flaky test lies, and the damage is not the failed run. It is that people stop believing red.

Once a suite has a handful of flaky tests, every failure gets a re-run before it gets read. At that point the suite has stopped being evidence, and a genuine defect can sit in a red pipeline for a week while everybody assumes it is the usual noise.

Flakiness is a trust problem that looks like a technical one.

The terms you will hear

• Flake rate. How often a test fails without a product change. Per test, over a window.
• Quarantine. Moving a test out of the blocking run while it is investigated.
• Retry. Re-running a failed test automatically, in the hope it passes.
• Order dependence. A test that only passes when another runs first.
• Race condition. Two things happening in an order the test did not expect.
• Hermetic test. One that creates everything it needs and depends on nothing outside itself.

The five causes

1. Timing. The test checks before the application has finished. Fixed waits are the usual culprit, and the usual "fix" that makes it worse on a slower machine.

2. Shared state. Two tests using the same customer, the same card, the same row. Passes alone, fails in parallel.

3. Test order. One test leaves state behind and another depends on it. Reordering or parallelising exposes it immediately.

4. Real randomness. Random data, a real clock, a live third party, a date that crosses midnight. The test is genuinely different each run, which is a determinism problem.

5. Environment. A slow container, a cold cache, a restarted sandbox. Not the test's fault, and still your report.

Why it matters

Because the cost is paid by the next real defect.

For example, a team with a 4 per cent flake rate re-runs most failed pipelines automatically. A genuine race condition in the basket appears as an intermittent failure and is assumed to be noise for three weeks. It reaches production, where it duplicates orders under load. Nothing about that story is a testing tool failing. It is trust, spent.

How to deal with it

1. Measure before fixing. Run the suite ten times on unchanged code, or read the last month of results. You want a flake rate per test, not an impression, which is exactly what run history is for.
2. Quarantine anything above your threshold immediately. For example, more than one failure in twenty on unchanged code. Out of the blocking run, ticket raised, owner named.
3. Diagnose by cause, not by test. Group the quarantined ones. Usually two or three causes explain all of them.
4. Replace waits with conditions. Wait for the element, the response or the state, never for a number of seconds.
5. Make each test hermetic. Its own customer, its own card, its own row. This removes causes two and three outright.
6. Control the clock, the seed and the network. Fix the sources of real variation rather than tolerating them.
7. Set a policy and hold it. Quarantine after N failures, fix or delete within two weeks, and never let the quarantine list grow silently.

A worked policy and clean-up

For example, here is one team's policy and what it found.

The grouping is the part worth copying. Nineteen flaky tests turned out to be five causes, and four of the five were fixed in bulk.

How to show you know it

• A measured flake rate. From twenty runs on one commit. It converts an argument into a number.
• A cause grouping. Nineteen tests, five causes. It shows you fix classes rather than instances.
• A written policy. Threshold, quarantine, deadline, review. Rare and immediately useful.
• A re-run rate. "Sixty per cent of pipelines were re-run, now under five." That is the trust metric.