Failure triage: clustering and log summarisation

Failure triage is working out what a set of failures actually means: how many distinct causes there are, which are real, and what to do first.

Two AI capabilities help, and both are unglamorous.

Clustering groups failures that share a cause. Two hundred red tests are rarely two hundred problems, and the grouping is the expensive part of the work.

Log summarisation turns forty thousand lines of output into the six that matter. Models are good at this because it is pattern recognition over text, which is exactly what they do.

Neither decides anything. Both remove the reading.

The terms you will hear

• Failure signature. The identifying detail of a failure: the assertion message, the exception, the first stack frame in your own code.
• Clustering. Grouping failures that share a signature or a cause.
• Root cause. The single change or condition producing the group.
• Noise. Failures caused by the environment or test data rather than the product.
• Quarantine. Removing a known-unreliable test from the blocking run while it is investigated.

Why it matters

Because the cost of a broken pipeline is mostly reading time, and reading time is what gets skipped under pressure.

For example, a nightly run comes back with 214 failures. The honest response takes two hours of scrolling. The usual response is to re-run it, see 190 failures, assume the environment, and merge anyway. Both cost more than they should, and the second one ships defects.

Clustering turns that into four groups in ten minutes. Three are one cause each, one is genuine noise, and the decision becomes obvious.

How to do it

1. Cluster mechanically first. Group by exception type, assertion message and the first stack frame in your own code. No model needed, and it resolves most runs.
2. Then ask a model to group the remainder. Paste the distinct failure signatures, not the whole log, and ask for candidate groupings with a reason for each.
3. Ask it to name the change. "Which of these failures could a change to the pricing service explain?" Models are good at this and confidently wrong sometimes, which is why the next step exists.
4. Verify one failure per cluster by hand. Open it, read the actual output, confirm the proposed cause. This is the step that makes the whole approach safe.
5. Separate noise explicitly. Environment, data, timeouts. Label them rather than fixing them, and check them against your run history to see which are chronic.
6. Report clusters, not counts. "214 failures, 4 causes, 1 blocking" is actionable, and it is the kind of number worth putting in a summary report.
7. Keep the clusters. A recurring cluster is a standing problem, and naming it is how it finally gets fixed.

A worked triage

For example, here is one nightly run, triaged in twelve minutes.

Cluster C is the point. It was 26 failures out of 214, buried under a 142-strong noise group, and it was the only one that mattered. Nobody scrolling for two hours reliably finds that.

How to show you know it

• A cluster report with causes. Four groups beats a count of 214 in every conversation.
• An invented link you caught. Showing where the model connected two unrelated groups proves you verify rather than accept.
• A recurring cluster you named. "This same timeout group appears every Tuesday" is how a chronic problem finally gets owned.
• A time figure. Twelve minutes against two hours is the argument for doing it this way.