QA Metrics: The Four Worth Reporting (and Why)
Every list of twenty QA metrics is eighteen you'll stop updating by March. Here are the four that survive one test: what decision changes when the number moves?
Twenty metrics, eighteen abandoned by March
Search for QA metrics and you'll find lists of twenty. Number of test cases written, pass rate, defects by severity, automation percentage, mean time to this, average that. It looks thorough. It's mostly theatre.
Here's what happens to that list of twenty. In January someone sets up the dashboard. By March, two of the numbers are still current and the rest are quietly stale, because nobody ever did anything differently because of them. That's not a discipline problem. It's a sign the metrics were never useful in the first place.
The useful question isn't "what can we measure?" Almost anything can be measured. It's "which few numbers, when they move, make us do something different?" This pillar argues for a short list — four metrics — and the single test each one has to pass to earn a place on it.
The one test every metric must pass
Before any number goes on a dashboard, ask one question: what decision changes when this moves?
If a metric goes up or down and your next action is exactly the same either way, it's not informing anything — it's decoration. "Number of test cases written" is the classic example. It climbs every sprint, it feels like progress, and it changes no decision at all. Whether it's 400 or 4,000 tells you nothing about whether you can ship.
A real metric is uncomfortable. When it moves the wrong way, it should point at something you now have to do — look harder at an area, hold a release, shift testing earlier. If you can't finish the sentence "when this gets worse, we…", drop the metric. It costs effort to maintain and hands you a feeling of rigour in return, which is worse than nothing.
A number that can't change a decision isn't a metric. It's a chart you maintain to feel measured.
The short list
Four metrics pass the test. Each one, when it moves, points at a specific action.
Escaped defects. The bugs that reached production — the ones your testing was supposed to catch and didn't. Sometimes called defect leakage or escape rate. When this rises, you look at what got through and why, and it changes what you test next. It's the closest thing QA has to a scoreboard, because it measures the actual miss, not the effort. A single embarrassing escape often teaches you more than a month of green dashboards.
Coverage of what matters. Not lines of code executed — whether the things that actually matter, the key requirements and risky areas, have tests behind them. When a critical area shows up uncovered, the action is obvious: write the cases before you ship it. This is the version of "coverage" that survives scrutiny, and the next section is about why.
Flakiness rate. How much of your green you can trust. When it climbs, your suite is turning into noise, and the action is to triage the unreliable tests before people start ignoring red altogether. A suite can be 95% green and still be worthless if you can't tell which green is real.
Time to detect. How long from a defect being introduced to being caught. When it grows, bugs are living longer before anyone notices, and the action is to push testing earlier and tighten the feedback loop. A bug caught in review is cheap; the same bug caught by a customer is not.
Why coverage percentage fails on its own
Coverage deserves a section of its own, because it's the metric most likely to lie to you while looking responsible.
Code coverage — the percentage of lines your tests execute — feels like it measures thoroughness. It doesn't, on its own. You can hit 90% line coverage and never once check the single business rule the feature exists to enforce.
The number counts how much of the code your tests ran, not whether you tested what the product is for. High coverage with the important rule untested is worse than an honest low number, because it buys false confidence.
What turns coverage into evidence is tying it to requirements instead of lines. "Coverage of what matters" asks a different question: of the things this release must do, how many have a test behind them? That number can change a decision, because it tells you exactly which critical behaviour is unguarded.
And it resists gaming in a way line coverage can't. You can pad line coverage with trivial tests that assert nothing important. You cannot fake having checked a rule you never wrote a case for — the requirement is either linked to a test or it visibly isn't.
Line coverage tells you how much of what you wrote, you ran. It never once saw the requirement you were supposed to build.
The metrics that don't make the cut
It's worth naming the numbers that usually top those lists of twenty, and why each one fails the single test.
Pass rate. The share of tests passing. It feels central, but it blends two different things — real failures and flaky ones — into one number, and on its own it rarely changes a decision. A 95% pass rate can hide a broken critical path or a pile of noise. Flakiness rate and escaped defects say what pass rate only gestures at.
Automation percentage. What fraction of your tests are automated. It measures activity, not value, and it nudges teams to automate the wrong things to move it. A high percentage of low-value automated tests isn't a quality signal — it's a maintenance bill dressed as one.
Number of test cases written. Pure output. More cases is not more quality; it's usually just more to maintain. Nothing you do changes when the number moves, which is the definition of theatre.
Mean time to resolve a bug. A useful delivery-flow metric, but it measures how fast you fix, not how well you test. It belongs on an engineering dashboard, not a quality one.
None of these are forbidden. Track one if it genuinely changes something for your team. But most get reported out of habit — and habit is exactly what fills a dashboard with numbers nobody acts on.
Goodhart: when a metric becomes a target
There's an old rule every QA leader should keep close: when a measure becomes a target, it stops being a good measure. Point a team at a number and they will move it — sometimes by doing better work, often by gaming it.
Watch how each of the four bends under pressure. Make escaped defects a target and production bugs get quietly reclassified as "enhancements," or the small ones stop being logged. Make coverage a target and trivial tests appear to pad the percentage.
The other two bend just as easily. Make flakiness rate a target and unreliable tests get deleted or quarantined to clean up the number — hiding the problem instead of fixing it. Make time to detect a target and investigations get closed early to stop the clock.
The defence isn't a cleverer metric; it's how you use it. Treat these numbers as evidence that starts a conversation, not as a score someone is graded on. Report them to yourself first, to decide where to look — not up the chain as a performance stat. The moment a metric becomes someone's grade, it stops telling you the truth.
Reporting to people who don't test
Most of the audience for these numbers has never written a test case and never will. A flakiness rate means nothing to them, and a coverage chart means less. The mistake is showing the chart naked and hoping the point lands.
Every chart you put in front of a non-tester needs one sentence in front of it, in their language, that says what decision it informs. Not "flakiness is at X" but "a chunk of our test signal is unreliable right now, so we're less sure than usual about this release — and here's what we're doing about it."
Do the same for the rest. Not "coverage of critical requirements is Y" but "these two important behaviours don't have tests yet, and that's the risk we'd be shipping." Translate every metric into a consequence for the thing they care about: whether to ship, what risk you're carrying, and what you're doing about it. The number is your evidence; the sentence is the message. Lead with the sentence, and the chart becomes support instead of homework.
Release readiness is your line, not a benchmark
Put the short list together and it becomes a release-readiness check: a small set of numbers you look at before shipping. Escaped-defect trend, coverage of the critical requirements, an acceptable level of flakiness, a time-to-detect that isn't growing.
One warning here. The threshold on each — what counts as "ready" — is a line your team draws, based on your product and your risk. There is no universal "good" number to import, and anyone selling you an industry benchmark for these is guessing. A payments platform and an internal tool should not share a bar. Set your own, write it down, and revisit it; a borrowed threshold is just someone else's guess wearing a suit.
Where the numbers come from
None of this computes without records, and this is where most metric programmes quietly fail. Each of the four needs something specific to have been captured in the first place:
- Escaped defects need defect records linked to the release they slipped through.
- Coverage of what matters needs cases traced to the requirements they verify.
- Flakiness rate needs run history — every run kept, not just the latest result.
- Time to detect needs timestamps: when a defect was introduced and when it was caught.
If your tooling overwrites results, or never links a case to a requirement, these numbers aren't just hard to compute — they're impossible, and no dashboard will conjure them from data you didn't keep. The metrics you can report are decided long before reporting, by what the underlying record actually stores.
This is the honest boundary for where a tool like Tesbo helps. Tesbo ties cases to the requirements they cover, so the "coverage of what matters" view is real. It does not compute your escaped-defect trend, flakiness rate, or time-to-detect for you today, and it doesn't run your tests — those come from your defect tracker, your run history, and your pipeline. Good reporting starts with recording the right things, wherever they live.
The short list, once more
Four numbers, each tied to a decision, each reported with a sentence a non-tester understands, each drawn from records you actually keep. That beats twenty metrics you'll abandon by March, every time.
The discipline isn't tracking more. It's the courage to cut everything that can't change a decision, and to report the few that can as evidence rather than theatre. A shorter list you act on is worth more than a long one you admire.
Questions people ask
What QA metrics are actually worth tracking?
A short list of four: escaped defects, coverage of what matters, flakiness rate, and time to detect. Each passes a single test — when the number moves, a specific decision changes. Metrics that fail that test, like "number of test cases written," look like progress but change nothing.
Is code coverage a good QA metric?
Not on its own. Line coverage measures how much code your tests ran, not whether you tested what the product is for — you can be 90% covered and miss the one rule that matters. It becomes useful only when tied to requirements, so it reflects coverage of critical behaviour rather than executed lines.
What's the difference between defect density and defect leakage?
Defect density is defects relative to size — per module or per thousand lines — and is mostly useful for spotting hotspots. Defect leakage, or escape rate, counts the bugs that reached production despite testing. For judging whether your testing is working, leakage is the more honest number, because it measures the actual miss.
What are good release readiness criteria?
A small set of thresholds on the short list — escaped-defect trend, coverage of critical requirements, acceptable flakiness, a stable time to detect — with the lines drawn by your team. There's no universal benchmark to import; a payments platform and an internal tool should set very different bars.
How do you report QA status to stakeholders?
Lead every chart with one sentence, in their language, that says what decision it informs — the risk you're carrying into the release and what you're doing about it. The metric is your evidence; the sentence is the message. Never show a number to a non-tester without translating it into a consequence.
