Flaky Test Detection: Finding Them in Your Run History

Flakiness doesn't show up in one run

A flaky test is one that passes and fails on the same code, for no good reason. Everyone has met one. The frustrating part is that you can't catch it in the act, because a single run tells you almost nothing.

Think of a slightly weighted coin. One flip looks completely normal — heads, fine. You only learn the coin is rigged by flipping it many times and noticing the pattern is off. A flaky test is the same. Flakiness isn't a property of one run; it's a property of the spread of runs.

Here's the good news: you almost certainly already have the data. Every test that runs in CI leaves a record. Detecting flaky tests is mostly a matter of reading records you're already keeping — and almost no team actually does it.

This post defines the measurement precisely enough that you can compute it against whatever run history your tooling keeps.

What flaky test detection actually measures

The definition is short: the same test, run against the same commit, producing different outcomes. Pass here, fail there, with the code held still. That disagreement is flakiness.

The word doing all the work is "same commit." If you compare a test's results across different commits, you're not measuring flakiness — you're measuring normal change. The code moved; of course the result might. A test that fails because someone broke the feature isn't flaky. It's correct.

So the whole measurement hinges on holding the code constant and watching the outcome wobble anyway. That means every run record has to know which commit it ran against. Miss that one field and you can't tell a flaky test from an honest failure — the two look identical in a list of pass/fail.

Window and threshold: what you catch vs what you drown in

Once you can group runs by commit, two choices decide what your detection actually surfaces.

The window is how far back you look — the last 20 runs, or the last two weeks. Too short, and one unlucky blip makes a solid test look flaky. Too long, and you're slow to notice new flakiness, while dragging in runs from code that has since changed.

The threshold is how much disagreement counts as flaky — fails one run in twenty, or one in five. Set it low and everything trips the wire; you drown in a list too long to act on. Set it high and only the worst offenders show, while the slow bleeders keep wasting an hour here and there.

There's no universal right answer, and anyone who hands you one number as "the standard" is guessing. Start loose, watch what the list looks like, and tighten until the top of it is things you'd genuinely stop to fix.

Pass rate and flakiness rate are not the same number

Most teams already report a pass rate, and quietly assume it covers flakiness. It doesn't, and the gap matters.

Pass rate lumps two very different failures together. A test that fails because the feature is genuinely broken lowers it. A test that fails half the time on unchanged code lowers it too. Same dent in the number, completely different problem — one is a real bug, the other is noise.

Flakiness rate isolates the second kind: the tests that can't make up their mind with the code held still. Reported side by side, the two numbers answer different questions, and you need both.

What your run records have to store

None of this is computable unless your run history keeps a few things on every single run:

• The test's stable ID — so "this test" means the same test across months, even if its title changed.
• The commit or build it ran against — the field that separates flakiness from honest change.
• The outcome — pass or fail at least; the error type helps.
• The timestamp — so you can apply a window at all.

Here's the trap that silently makes flakiness un-measurable: storing only the latest result on the test itself. Overwrite the last outcome each run and you have exactly one data point per test — and one point can never show a distribution. You threw the history away before you could read it.

This is why runs have to live as their own records, separate from the case. A tool that keeps every run instead of overwriting it is what makes detection possible in the first place. Tesbo keeps that execution history as first-class records; computing and ranking flakiness from it is work you do, not something it does for you today.

Reading the ranked list: a work queue, not a leaderboard

Compute a flakiness rate per test, sort descending, and you get a ranked list. The important thing is what that list is for.

It's a work queue, not a wall of shame. The top few tests are where your trust is leaking fastest, so that's where you start — reproduce, classify, then quarantine or fix. You don't attack the whole list at once; you take the top, deal with it, and let the list re-rank.

And keep one possibility open as you go: sometimes the test is fine and the case behind it is the problem. An expected result vague enough to read two ways will produce two outcomes on identical code — and that looks exactly like flakiness. When the fix isn't in the code, it's in the case: tighten the expected result so there's only one honest answer.

What detection is, and isn't

Detection is measurement. It tells you which tests you can't trust and in what order to deal with them. It does not fix anything, and it won't hand you a threshold that's right for every suite — you tune that to your own noise.

Done honestly, though, it turns a vague, shared complaint — "the suite is flaky" — into a specific, ranked list of tests, each with the data to prove it. That's the whole point: you can't fix what you refuse to measure.