Incident response, RCA and postmortems

When production breaks at 2am, most QA engineers assume there is nothing useful for them to do until the fire is out and someone writes a ticket. That assumption wastes the person on the team who best understands how the system is supposed to behave. Testers who show up during the incident, not just after, shorten the outage and make the resulting root cause analysis far more useful.

What a tester actually does during a live incident

During an active incident, the loudest voice in the room is usually guessing. A tester's job is to bring evidence instead. Reproduce the reported symptom against a known-good baseline, confirm which environment and which recent deploy the behavior appears in, and rule out red herrings fast.

This narrows the search space for the engineers doing the actual fix, which is the single biggest lever on mean time to restore.

Good incident testers also know the difference between confirming a symptom and confirming a cause. Saying "yes, checkout is returning 500s for EU customers" is useful triage information. Claiming "it's the payment gateway" without evidence just adds a wrong hypothesis to a room that already has too many, especially when the team is also fighting flaky tests that make "confirmed" hard to trust. Stick to what you can verify directly and say plainly when something is a guess.

Running the postmortem: blameless does not mean toothless

A blameless postmortem separates the person from the mistake, on the well-founded premise that individuals rarely fail alone. They fail inside a system of gaps in tooling, process, and information that let a small error become a big one. But blameless is often misread as "we don't assign responsibility for fixing anything," which produces a document nobody acts on.

The fix is to separate two questions clearly in the postmortem format: what happened (a blameless, factual timeline) and what changes (a set of owned, dated action items). The timeline can be honest about who did what without it turning into a performance review, because the point of writing it down is to find the systemic gap, not the individual lapse.

The five whys, and where it breaks down

The five whys technique is useful for surfacing a chain of causes quickly, but it breaks down when the answer at each step depends on who is answering. Consider a team at a logistics SaaS company that had a shipping label generation outage. Their first pass at five whys concluded "the root cause was a developer forgetting to update a config value," which is true but useless, because it does not explain why nothing caught the mistake before it reached production.

Redoing the exercise with a QA lead in the room extended the chain. The config wasn't validated at deploy time. The staging environment used different config values than production, so the mistake was invisible there. There was also no synthetic check hitting the label generation endpoint that would have caught the failure within minutes instead of hours. The actual fix was a config validation step and a synthetic check, not a reminder to be more careful.

Writing action items that survive contact with next sprint

An action item that says "improve testing around payment flows" will still be open a year later, because nobody can tell when it is done. Every action item from a postmortem needs an owner, a date, and a description specific enough that a stranger could tell whether it was completed. Compare the two:

• Vague: "Add more monitoring for checkout."
• Specific: "Add a synthetic check hitting POST /checkout/submit every 2 minutes, alerting if p95 latency exceeds 3s or error rate exceeds 1%. Owner: platform team. Due: next sprint."

Tracking whether postmortems actually change anything

The postmortem process itself needs a metric or it decays into a formality. Track the percentage of action items closed within their due date, and separately track repeat incidents (the same root cause showing up again within a quarter). A rising repeat-incident rate means the RCA process is documenting problems without fixing them, regardless of how thorough the documents look. Pair this with your escape rate trend, since a rising repeat-incident count usually shows up there first.

FAQ