The Flaky Test Triage Runbook: What To Do In The First 30 Minutes

The build just went red and nobody touched the code that broke. It's 4:45 on a Thursday, the release is scheduled for tonight, and someone on the team has to decide in the next few minutes whether this is a real bug or noise. Most teams handle this by gut feeling: someone reruns the job, it passes, everyone shrugs and moves on. That's how a genuine defect slips into a Thursday release wearing a flaky test's clothes. This is a runbook, not an essay on why flakiness is bad. Follow it in order the next time a build goes red for no obvious reason.

Step one: does it fail alone, or only in the suite

Before anything else, rerun the failing test by itself, isolated from the rest of the suite. This single question splits the investigation in half.

If it fails alone, the problem lives inside that test or the thing it's testing. You're looking at timing, a bad assertion, or a real regression. If it only fails inside the full run, the problem is almost always something the suite is doing to it: shared state, leftover data, or execution order. Skipping this step is the single biggest reason triage drags on for hours instead of minutes.

Classifying the failure

Once you know whether it's isolated or suite dependent, classify what's actually going wrong. Each type has a recognizable tell.

• Timing: the test waits a fixed 2 seconds for something that sometimes takes 3, and fails only under load
• Shared state: two tests both write to the same in-memory cache, and whichever runs second sees stale data
• Order dependence: the test passes when run after test B, fails when run after test C, because B happens to reset something C doesn't
• Data collision: two parallel test runs create a user with the same email and one insert fails
• Environment: the test hits a staging API that's slow or down, and the failure has nothing to do with the code at all

Write down which one it looks like before moving on. That single word saved in a ticket or a chat message is what lets someone else pick up the investigation later without starting from zero.

Quarantine now, or fix now

The decision here has to be fast, because a red build is blocking someone. The rule: if you can name the cause with confidence in under 15 minutes, fix it now. If you can't, quarantine it and move on.

Quarantining means marking the test so it doesn't block the build, while still running it and logging the result somewhere visible. A quarantined test should always get an expiry, something like 5 working days, after which it either gets fixed or gets deleted. A test that's been quarantined for two months with nobody looking at it is worse than no test at all, because it's actively teaching the team to ignore red.

What to record so nobody repeats the work

The investigation you just did is worthless to the next person if it lives only in your head. Record four things in the ticket or the test's own notes:

• The classification from the step above (timing, shared state, order, data, environment)
• What you actually tried, even the things that didn't work
• The exact conditions that reproduce it, like "only when run after the payment suite"
• The quarantine expiry date, if you quarantined it

This takes two minutes and saves someone else the same 30 minutes you just spent, the next time this test flakes again.

Closing the loop: the test case might be the thing that's wrong

Here's the step most runbooks skip. Sometimes the code is fine, the test's automation is fine, and the actual test case behind it describes the wrong behaviour. A login test that expects a session to last exactly 15 minutes might be flaky because the requirement changed to 20 minutes and nobody updated the case.

Before you close the ticket, ask whether the case itself still describes something true. If the answer is no, the fix isn't in the test code at all. It's in the case, and skipping that check is how the same test goes flaky again in three weeks for a completely different reason.