Flaky Tests Are a Design Problem, Not Bad Luck
A flaky test isn't bad luck. It assumed something about timing, state, or isolation the system never guaranteed. A retry hides that — it doesn't fix it.
A retry is a decision to ship an unknown
Most teams treat flakiness like weather — a fact of life you put up with and suppress. The test failed for no reason, so you rerun it. It's green now, so you move on. Raise a timeout here, add a retry there, and the board goes back to green.
But a flaky test is not random, and it is not bad luck. It's a test that made an assumption about timing, state, or isolation that the system it runs against never actually guaranteed. When that assumption holds, the test passes. When it doesn't, the test fails. That's not noise; that's a design bug — in the test.
Which is why a retry doesn't fix anything. The assumption is still there. All a retry does is stop telling you about it. Your suite still fails sometimes, on the same faulty assumption; you've just muted the alarm and called it a fix. This pillar is about treating flakiness as the design problem it is: naming the patterns, finding the assumption, and removing it.
The six patterns
Almost every flaky test falls into one of six patterns. For each, there's a tell in the run log, an experiment that confirms it, and a fix that removes the bad assumption rather than muting it.
1. Implicit waits and timing
Symptom: the test fails intermittently with "element not found" or an assertion that ran a beat too early, and it fails more often on a busy CI machine than on your laptop.
Experiment: slow the machine down, or add load, and watch the failure rate climb. The test is racing the application and losing when the app is a little slower than usual.
Fix: wait for the condition, not the clock. Wait until the element exists or the state is reached, rather than sleeping a fixed number of milliseconds and hoping. The bad assumption was "the app is ready instantly." Replace it with "proceed when the app is actually ready."
2. Shared state between tests
Symptom: the test passes when you run it alone and fails inside the full suite. Whether it fails depends on what ran before it.
Experiment: run it in isolation — green. Run it right after some other test — red. Something the earlier test left behind is bleeding into this one.
Fix: make each test set up and tear down its own state, and stop leaning on global or shared state that another test can change. The assumption was "the world starts clean." Make it true for every test, every time.
3. Order dependence
Symptom: the suite passes in one order and fails in another. Change which tests run first and different tests break.
Experiment: randomise the test order and run it a few times. New failures appear that were hidden by the old, lucky ordering.
Fix: remove the dependencies between tests so each one stands alone. It's a cousin of shared state, but the tell is specifically ordering — test B only passes because test A happened to run first and leave the right thing behind.
4. Test data collisions
Symptom: the test fails when run in parallel, or when two people run it at once, but passes when run serially. Two tests are fighting over the same record.
Experiment: run the suite in parallel and the collisions appear; run it serially and they vanish. Two tests are using the same fixed username, account, or row and stepping on each other.
Fix: give each test its own unique, isolated data — generated or namespaced per run — instead of a shared fixture everyone reuses. The assumption was "I'm the only one touching this record." In parallel, that was never true.
Half of flakiness is a test quietly assuming it's the only thing in the room — the only one using that record, that order, that moment in time. Run it alongside everything else and the assumption breaks.
5. Environment drift
Symptom: it passes locally and fails in CI, or starts failing after an environment change. Time zones, locale, a system clock, or an external service are involved.
Experiment: reproduce the environment difference and the failure follows the environment, not the code. Switch the locale or the time zone and the behaviour changes.
Fix: control the environment the test depends on. Pin the locale and clock, and mock or control external services rather than reaching out to a live one. The assumption was "everywhere is like my machine." Make the test's environment explicit and stable.
6. Assertions on things the system never promised
Symptom: the test fails on the ordering of results the API never guaranteed, on a timestamp, or on some value that's legitimately non-deterministic.
Experiment: check the spec. Did the system ever actually promise this order, this exact value, this timing? Often the honest answer is no.
Fix: assert only on behaviour the system guarantees. Here the test is simply wrong — it's checking something the product never committed to, so a passing run was luck, not correctness. Tighten the assertion to what's actually promised.
Detection: flakiness lives in the distribution
Notice what all six patterns have in common: you can't see any of them in a single run. One run is one data point, and one data point can't show that a test is unreliable. This is why the most common flaky-test reflex is also the weakest evidence there is.
"It failed, I reran it, it passed, so it's fine" proves nothing. A single re-run that goes green tells you the test can pass, which you already knew. It says nothing about how often it fails or why. Flakiness is a property of the distribution — the same test, on the same commit, producing different outcomes across many runs — and you can only see it by looking across that history, not at the last result.
That's what a flakiness rate measures: over a window of recent runs on unchanged code, how often does a test disagree with itself? Ranked by that number, your tests become a work queue, with the least trustworthy at the top. But none of it is computable if you only ever keep the latest result — you need the run history to see the shape at all.
"It passed on retry" is not a diagnosis. It's the sound of an alarm being silenced. The test still fails sometimes; you've just decided to stop counting.
Quarantine is a state, not a mute button
When a flaky test is blocking everyone, pulling it out of the blocking suite is reasonable. Done right, quarantine is a deliberate, time-boxed state: the test still runs and reports, it just stops failing the build for everyone while one person fixes it. Done wrong, quarantine is where tests go to die.
The difference is two attributes. Every quarantined test needs an owner — a named person responsible for it — and an expiry — a date by which it's fixed or deleted, not silently renewed. With those, quarantine is a hospital bed: a temporary place to heal. Without them, it's a graveyard, and the coverage that test represented is quietly gone while everyone assumes it's still protecting them.
The failure mode is a quarantine list that only grows. Tests get muted, nobody owns them, no date ever comes, and a year later you have a pile of disabled tests and no idea which real gaps they left behind. If your quarantine has no expiry dates, it isn't quarantine — it's a delete you're too polite to commit.
Retries: the narrow case where they're legitimate
Retries have a bad reputation for a good reason — most of the time they're the anti-pattern above, muting a design bug you should fix. But there is a narrow, honest case for them, and it's worth stating precisely so it isn't used as cover for the rest.
The legitimate case is a genuinely non-deterministic dependency you don't control — a third-party sandbox that occasionally hiccups, a service with a known, rare, unavoidable blip. There, a bounded retry is a pragmatic mitigation rather than a lie. But it comes with a condition: accounting. The retry has to be recorded and counted, so a rising retry rate is visible and someone notices when "rare" becomes "constant."
A retry with no accounting is just a silent mute, and it's how a suite rots without anyone deciding to let it. The moment retries happen quietly, your suite is failing and not telling you, and the number of hidden failures only grows. So: retry rarely, retry bounded, and always count it — a retry you can't see on a dashboard is a failure you've agreed not to know about.
What a suite nobody trusts costs
The real cost of flakiness isn't the wasted minutes spent re-running. It's much worse, and it's measured in shipped bugs.
When a suite is noisy, a genuine failure looks exactly like the usual flakiness. The test goes red, someone shrugs, reruns it or waves it through as "probably flaky," and a real defect walks straight past the check that existed to catch it.
The suite cried wolf so many times that the one true warning got ignored. Every false alarm spends a little of the team's trust, and once that trust is gone, the suite stops functioning as a safety net even while it's still running.
That's why a suite nobody trusts is worse than no suite at all. No suite gives you no confidence, and you act accordingly. A flaky suite gives you false confidence — a green board you've quietly learned to disbelieve — and false confidence is what ships the bug. Trust is the entire asset a test suite provides. Flakiness spends it, one shrugged-off red at a time.
When the case is the problem, not the code
One more source of apparent flakiness deserves a mention, because it hides in plain sight: sometimes the automated test is fine and the case behind it is wrong.
A test case with a vague expected result — "the page loads correctly," "an error is shown" — can be read two different ways by two runs, or by the same run on a slightly different day.
Two legitimate outcomes from one ambiguous specification looks identical to flakiness in the log, but the fix isn't in the timing or the isolation. It's in the case: write a single, unambiguous expected result so there's only one honest answer. This is where a clear, reviewed case record removes a whole class of "flakiness" before it starts.
It's worth being clear about what a tool does and doesn't do here. Tesbo keeps the case record and the run history that let you write unambiguous cases and measure flakiness in the first place. It does not detect, retry, quarantine, or fix flaky tests, and it doesn't run your suite — the framework and CI runner stay yours. The record is the raw material; the diagnosis and the fix are engineering work.
Flakiness is a design problem
Pull it together and the frame is simple. Every flaky test encodes an assumption the system never guaranteed — about timing, shared state, order, data, environment, or a promise the product never made. The six patterns are just the six shapes that assumption tends to take.
Retries and timeouts hide the assumption. They don't remove it, and they quietly cost you the one thing a suite exists to provide: trust. Treat a flaky test as a design bug in the test, find the assumption it's making, and fix it — and you get back a green board that actually means green. A suite you can trust is worth far more than a suite that merely looks passed.
Questions people ask
Why do tests fail randomly?
They don't, really — they fail when an assumption the test made stops holding. A flaky test relies on something the system never guaranteed: that the app is ready instantly, that it's the only thing using a record, that the environment matches your laptop. When that assumption breaks, the test fails, and it looks random only because the assumption is hidden.
What causes flaky tests?
Six patterns account for most of it: implicit waits and timing, shared state between tests, order dependence, test data collisions, environment drift, and assertions on things the system never promised. Each has a distinct tell in the run log and a fix that removes the faulty assumption rather than hiding it behind a retry.
Are test retries a good idea?
Usually not — a retry most often mutes a real design bug instead of fixing it. The narrow legitimate case is a genuinely uncontrollable external dependency, and even then only with accounting: the retry must be recorded and counted so a rising retry rate is visible. A silent retry is just a failure you've agreed not to see.
How do you do flaky test root cause analysis?
Start with one experiment: does it fail in isolation or only in the suite? That splits the causes in two. From there, classify by pattern using the symptom and a confirming experiment — slow the machine for timing, randomise order for order dependence, run in parallel for data collisions — then apply the fix for that pattern rather than a blanket retry.
What does a flaky test suite actually cost?
Its worst cost is shipped defects. When failures are usually noise, a real failure gets dismissed as "probably flaky" and the bug reaches production. A suite nobody trusts is worse than no suite, because it hands you false confidence — a green board you've learned to disbelieve — and false confidence is what lets the genuine failure through.
