The QA Metrics That Survive Contact With A Dashboard

A head of quality picks up a metric on a Monday and by Thursday the team has quietly reorganized its week around hitting it. That is not a hypothetical. It happens to test pass rate, to coverage percentage, to almost anything with a number attached and a name in front of leadership. The usual advice at this point is to measure systems, not people, which is true and also not something you can act on before your next standup. This post gives you the actual test. It is one property, it takes ten minutes to run against a metric you already report, and it tells you whether that metric will survive being looked at by someone with an incentive to look good.

The rule in one sentence

A metric is hard to game when it comes from a record nobody edits by hand. A metric is easy to game when someone types it into a status report.

That is the whole rule. Test pass rate that a person can round up before the Friday sync is soft. A count of test runs pulled straight from execution history, with no manual step between the run and the number, is hard to move without actually changing what happened. The difference is not sophistication or intent. It is whether a human touches the number between the event and the report.

This is the missing half of the usual "measure systems not people" advice. That framing tells you what to avoid. It does not tell you which of the metrics already on your dashboard are safe to keep reporting and which ones are quietly training your team to perform for the number instead of doing the work.

Running the four metrics you already have through the test

Most quality dashboards carry some version of these four. Run each one through the rule above.

• Test pass rate. If it is computed from the CI run log, it is derived. If someone marks flaky failures as "not a real bug" before the report goes out, it is edited, and it will drift toward 100% over a quarter without the product getting any safer.
• Escaped defect count. This one is close to unfakeable if it is sourced straight from the incident tracker. The number of bugs a customer found in production is a fact about the world, not a self report, unless someone starts quietly reclassifying incidents as "not a defect."
• Coverage percentage. This is the one that looks derived and is not. A team can hit 90% line coverage by writing tests that execute code without asserting anything meaningful about it. The number comes from a tool, but the tests behind it are still handwritten, so the incentive to game survives the automation.
• Cycle time from commit to test result. Derived from timestamps in the pipeline, hard to move without genuinely working faster, and one of the few metrics on this list that tends to hold up under pressure.

Two of these pass cleanly. Two look automated but still route through a human decision that can bend under pressure. That is the actual audit, and it is worth running against whatever four your own dashboard carries, because the answer is rarely the same team to team.

The two that still fail, and why you watch them anyway

Coverage percentage fails the test even when the tool computes it, because the input to the tool is still a human decision about what to test. Escaped defect count mostly passes, but the classification step ("is this really a defect, or a duplicate, or a support question") is a soft spot a team under deadline pressure will lean on.

Neither belongs on a leadership dashboard as a target. Both are worth watching privately, on your own screen, not the shared one. A coverage number that drops sharply after a big refactor is a useful signal to a head of quality even though it would be a bad target for the team writing the tests. The distinction is between a metric you monitor for yourself and a metric you hand someone else as a goal. The first can be soft. The second cannot.

What to say when leadership asks for a metric that will get gamed

This conversation happens to almost every head of quality eventually, usually after a slide with "95% test pass rate" gets circulated and somebody upstairs decides that number should be a target for the next quarter. Here is a version of it that works.

They ask: "Can we set a target of 95% pass rate for the team this quarter?"

You answer: "We can report it, but I'd rather not target it. The team controls the number by skipping or deprioritizing flaky tests, not by shipping fewer bugs. If you want a target, escaped defect count from the incident tracker moves for the same reason the product actually got safer, not because someone reclassified a failure."

That reframes the ask without refusing it. It also gives leadership a metric to hold instead of a flat no, which is usually what makes the conversation land instead of stall.

What this changes about your next report

The practical shift is small. Before you add a number to next month's report, run it through the one question: is this derived from a record nobody edited, or typed in by someone with a reason to round up? Keep the first kind on the shared dashboard. Keep the second kind, if you keep it at all, on your own screen where it can be a signal instead of a target.

For the fuller treatment of which metrics to report and why, see our QA metrics post. For the argument behind why an unedited record is the thing that makes a metric trustworthy in the first place, see our post on audit trails.