Quality metrics: DORA, escape rate, MTTR

Ask five engineering leaders what "good quality" means in numbers and you will get five different dashboards, most of them measuring effort instead of outcome. Test count, coverage percentage, and bugs found per sprint feel like quality metrics but mostly measure how busy the QA team looked. The DORA metrics, escape rate, and mean time to restore are different: they describe what a customer actually experienced, and they are much harder to game without it showing up somewhere else.

Why test count and coverage percentage are the classic own goals

Coverage percentage rewards writing tests, not writing tests that catch bugs. A team under pressure to hit 80% coverage will write shallow assertions against getters and setters, inflating the number while leaving the actual risky logic untested. The metric goes up, quality does not move.

Test count has the same problem in a different shape. Nobody has ever shipped a more reliable product because the suite grew from 2,000 tests to 4,000. If those 2,000 new tests are duplicated coverage of the same happy path, they add runtime and maintenance cost without adding a single caught defect. A metric that can be satisfied without improving the thing it claims to measure will eventually be satisfied that way, because that is always the cheaper path.

The four DORA metrics

DORA (DevOps Research and Assessment) settled on four measures that correlate with both delivery speed and stability, based on years of survey data across thousands of teams:

• Deployment frequency: how often code reaches production. Elite teams deploy on demand, multiple times a day.
• Lead time for changes: the time from a commit landing to it running in production.
• Change failure rate: the percentage of deployments that cause a failure requiring a fix, rollback, or patch.
• Time to restore service: how long an incident takes to resolve once it starts.

The first two describe speed. The last two describe stability. The insight that made DORA influential is that these are not a tradeoff. The highest-performing teams are fast and stable at the same time. That happens because the practices that speed up delivery, like small changes and fast feedback tied into CI/CD for test suites, are the same practices that make failures smaller and easier to fix.

Escape rate: the number that tells you if testing worked

Escape rate is the percentage of defects found in production versus defects found before release. It answers the one question coverage percentage cannot: did the testing that happened actually work? A team can have 90% code coverage and a terrible escape rate if the tests exercise the wrong paths.

Consider a payments team at a mid-size fintech that tracked escape rate quarterly. They had strong coverage numbers and a large regression suite, but escape rate crept up two quarters running.

Digging into the escaped defects showed a pattern. Nearly all of them involved currency conversion edge cases that the test data generator never produced, because the seed data used round dollar amounts. The fix was not more tests, it was better test data covering fractional currency conversions and rounding boundaries. Escape rate dropped the following quarter without the coverage number changing at all.

MTTR and why it matters more than time to first bug

Mean time to restore (MTTR) measures how long a real user is affected once something breaks, from detection to resolution. It matters more for the business than how many bugs your team found in staging, because staging bugs never affected a customer. A one-hour production outage caught and fixed fast can cost less in real terms than a subtle bug that sat undetected for a month.

Good MTTR depends on things testers can directly influence: clear incident response processes, dashboards that surface the right signal fast, and rollback mechanisms that have actually been rehearsed rather than assumed to work. A team that only measures pre-release quality is blind to half of what determines whether users notice a problem.

Building a scorecard that resists gaming

The safest metrics to defend in a leadership review are ratios and outcomes rather than raw counts. A raw bug count invites arguments about severity, a change failure rate does not, because it is normalized against how many deployments happened. Track escape rate and change failure rate together, since a team can suppress one by inflating pre-release defect logging unless both are watched. Watching flakiness alongside these numbers matters too, since a flaky suite makes any pre-release defect count unreliable.

FAQ