Software Quality Metrics That Actually Predict Failure

There's a specific kind of engineering dashboard that should worry you: the one where every number is green and the team still had two incidents last month. Test count up. Coverage at 84%. Pass rate 98.7%. Bugs closed exceeding bugs opened. And yet customers keep finding problems first.

That dashboard isn't lying. It's measuring the wrong things — reporting activity when what leadership needs is risk. Software quality metrics are supposed to answer one question: how likely is it that our next release hurts a customer? Most of the numbers teams collect cannot answer that question even in principle.

This article separates the two. What a quality metric is actually for, which popular metrics fail the test, the handful that genuinely predict failure, and how to read them together without creating the perverse incentives that ruin measurement programs.

What a quality metric is for

A metric earns its place on a dashboard if it does at least one of two things:

• Predicts — it moves before a failure, giving you time to act.
• Diagnoses — it tells you where to look when something has gone wrong.

Anything that does neither is decoration. It might describe effort accurately and still be useless for decisions, and dashboards full of accurate-but-useless numbers are worse than no dashboard: they create the feeling of oversight without the substance.

The second constraint is Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Every quality metric can be gamed, usually without anyone intending to game it. Target test count and you'll get many trivial tests. Target coverage and you'll get assertion-free tests that execute lines. Target bug closure rate and you'll get bugs closed as "cannot reproduce."

The popular metrics that don't predict anything

Number of tests

Measures how much test code exists. Says nothing about what it verifies. A suite of 6,000 tests that never exercises checkout is worth less than 30 tests that do. The number rises steadily and reassuringly while risk does whatever it likes.

Code coverage

The most over-trusted number in software. Coverage tells you which lines executed during a test run — not whether behavior was checked. A test with no assertions produces full coverage. Coverage is genuinely useful in one direction only: uncovered code is definitely untested. Covered code is merely possibly tested. We unpacked this at length in Code Coverage Is Not Quality. If you want a coverage-shaped metric with teeth, mutation testing — deliberately introducing faults to see whether tests notice — measures something coverage cannot.

Overall pass rate

A 98% pass rate sounds excellent and is nearly uninterpretable. Which 2% failed? If it's the same two flaky tests every run, the number is noise. If it's checkout, the release should be blocked. An aggregate that mixes critical and cosmetic tests hides exactly the information you need.

Bugs opened and closed

Bug counts measure reporting behavior as much as software behavior. A quiet month can mean a stable product or a disengaged support team. Worse, the metric usually treats a data-corruption bug and a misaligned icon as equal units.

Manual test cases executed

Measures human effort spent, not risk retired. It reliably goes up as the manual regression burden grows — which is a cost signal, not a quality one.

The metrics that earn their place

Six or seven numbers, well chosen, tell you more than thirty. Here are the ones that consistently predict or diagnose failure.

1. Escaped defect rate

How many defects reached production and were found by someone other than your tests, per release or per month — segmented by severity.

This is the closest thing to a direct measurement of quality, because it counts the failures of your entire quality system rather than the activity inside it. Track it as a rate against release volume (escapes per release), not as a raw count, or it will simply track how much you shipped.

Segment by severity, always. Twelve cosmetic escapes and one payment escape are different months.

2. Mean time to detect (MTTD)

How long a defect lives in production before anyone knows.

The most underrated metric in the entire discipline. Because the cost of a production bug scales with how long it's live, MTTD is a direct multiplier on the damage of every escape. It's also the metric teams almost never collect — most bug trackers record when a bug was reported, not when it was introduced.

A useful proxy if you have nothing: for each production bug, record the deploy that introduced it and the timestamp it was first noticed. The gap is your MTTD.

3. Change failure rate

The percentage of deployments that cause degraded service and require remediation — a rollback, hotfix, or patch.

One of the four key metrics from Google Cloud's DORA research program, and the one most directly about quality. Its virtue is that it can't be gamed by shipping less: it's a ratio. A team that deploys 200 times a month with a 3% change failure rate is in dramatically better shape than one deploying 4 times a month with 25%.

4. Time to restore service

How long from "something is wrong in production" to "customers are fine again."

The other DORA metric that belongs on a quality dashboard. Paired with MTTD it gives you the full incident timeline: detection plus recovery equals total customer pain. Teams frequently optimize this one because it's visible, while ignoring MTTD — which is often the larger half.

5. Critical-flow coverage

What percentage of your critical user journeys are verified automatically, before release and in production.

The metric that should replace code coverage on leadership dashboards. It has the properties you want: it's expressed in business terms, it's hard to game (you can't fake a passing checkout test), and it maps directly to consequence. If you have 14 critical journeys and 9 have automated end-to-end verification, your number is 64% — and everyone in the room understands what the missing 36% means. Building that denominator honestly is the work described in Critical User Journeys.

6. Flake rate

The percentage of test failures that turn out not to be real defects.

This measures the trustworthiness of your signal, which gates the value of every other test metric. Once flake rate passes roughly 5%, engineers begin re-running rather than reading failures, and the suite stops functioning as a control. Google's engineering teams have written extensively about this dynamic, and it applies at every scale: an untrusted test suite is functionally a disabled one. See Flaky Tests.

7. Rework ratio

The share of engineering time spent fixing recently shipped work rather than building new work.

The clearest translation of quality into a language the business already speaks. When rework rises, velocity is being spent maintaining the illusion of past velocity. It's also the metric that makes the case for quality investment without any borrowed multipliers.

Read them in pairs, never alone

Single metrics mislead. Pairs are much harder to fool, because improving one at the expense of the other shows up immediately.

• Deployment frequency × change failure rate. Speed without stability is recklessness; stability without speed is stagnation. Neither number means anything alone. This pairing is the core insight of the DORA research.
• MTTD × time to restore. Together they describe the full duration of customer pain. A great recovery time attached to a terrible detection time means you're excellent at fixing things you find late.
• Critical-flow coverage × escaped defect rate. If coverage is high and escapes are still frequent, your flow inventory is wrong — you're protecting the wrong journeys. That's a far more actionable conclusion than "we need more tests."
• Flake rate × pass rate. A 98% pass rate with a 10% flake rate is not a 98% pass rate. It's an unknown pass rate.

And prefer distributions to averages. Average time to detect hides the six-week bug in a pile of ten-minute ones. Report p50 and p90; the tail is where the expensive incidents are.

How teams break their own measurement

Measuring individuals. Bugs-per-developer or tests-written-per-engineer reliably produces defensive behavior: less risky work, quieter reporting, gamed numbers. Quality is a property of a system, and measuring it per person destroys the honesty the data depends on.

Turning every metric into a target. Watch most metrics; put targets on very few. A target on escaped defects is reasonable. A target on coverage produces assertion-free tests within a quarter.

Counting without a denominator. "We had 12 escapes" is meaningless without release count, traffic, or scope. Rates are comparable across time; counts are not.

Reporting metrics nobody acts on. If a number has never once changed a decision, delete it. Every unused metric on a dashboard makes the used ones harder to see.

Ignoring severity. Any quality metric that treats all defects as equal units will eventually be optimized by fixing easy ones.

How AI changes measurement

Two shifts are worth planning for.

Metrics get cheaper to collect. Correlating a production error to the deploy that introduced it, clustering 200 red tests into three root causes, classifying an escape by severity and affected flow — this is exactly the sort of judgment-under-messy-data work that models now handle competently. Metrics that were theoretically ideal but practically uncollectable (MTTD is the canonical example) become routine.

Some metrics lose meaning. When tests are generated rather than hand-written, test count and coverage become nearly free to inflate — a model can produce 10,000 tests and 95% coverage in an afternoon. Any metric measuring the volume of testing artifacts is now essentially worthless as a quality signal. The metrics that survive are the outcome-based ones: escapes, detection time, change failure rate, flow coverage. This is one more reason the shift described in How AI Is Changing Software Testing in 2026 is structural rather than cosmetic.

A third, quieter shift: with continuous verification running against production, quality stops being a quarterly report and becomes a live reading. The question changes from "how did we do last quarter?" to "are our critical flows working right now?"

Where BuniOD fits

Most of the metrics above are hard to collect not because the math is difficult but because the underlying inventory doesn't exist — you can't report critical-flow coverage without knowing what your critical flows are, and you can't compute MTTD if nothing is checking production between customer reports.

BuniOD produces both as a side effect of how it works: you describe the business flows that matter in plain language, it generates and runs the scenarios continuously before and after release, and it reports which flows are currently verified and which are failing. That turns "critical-flow coverage" and "time to detect" from aspirational metrics into numbers that are simply available.

Conclusion

Most quality dashboards measure how much testing happened. The useful ones measure how much risk remains.

If you rebuild yours around one principle, make it this: report outcomes, not activity. Concretely:

• Replace code coverage with critical-flow coverage.
• Replace bug counts with escaped defect rate by severity.
• Add mean time to detect — the metric most likely to be both terrible and unmeasured.
• Watch change failure rate and time to restore together, as DORA intends.
• Track flake rate, because it determines whether any other test metric can be trusted.

Five numbers. Each one answers a question a customer would recognize, and none of them improves unless the software genuinely got safer to ship.