Code Coverage Is Not Quality: Measure What Actually Matters

Code coverage is the most reported and least understood number in software quality. A team announces "we're at 87% coverage" the way a factory reports uptime — as if the figure, on its own, means the product works. It does not. Coverage measures which lines of code executed while your tests ran. It says nothing about whether the behavior those lines produce is correct.

This distinction is not academic. It is the difference between a suite that protects you and one that gives you a false sense of safety. You can reach 100% coverage with tests that assert nothing meaningful, and you can catch a critical regression with a suite that leaves half the codebase untouched. The number and the outcome are only loosely related.

This article is about what coverage actually measures, the specific ways it misleads teams, and the metrics that track quality more honestly.

What coverage actually measures

"Code coverage" is an umbrella over several distinct metrics, and conflating them is the first mistake:

• Line coverage — the percentage of lines executed at least once. The most common and the weakest.
• Branch coverage — whether each side of every if/else was taken. Stronger, because a line can run without both branches being tested.
• Function coverage — whether each function was called at all.
• Condition coverage — whether each boolean sub-expression evaluated both true and false.

All of them share one property: they measure execution, not verification. A line is "covered" the instant a test causes it to run — even if the test never checks the result.

// This test gives the function 100% line coverage.
test("calculates total", () => {
  calculateTotal([{ price: 10 }, { price: 20 }]); // called, so "covered"
  // ...but there is no assertion. A total of NaN would pass.
});

That example is the whole problem in miniature. Coverage went up; confidence did not move at all. Multiply this across a large suite written under pressure to "hit the number," and you get the paradox teams live with every day: a high coverage figure and a steady stream of production incidents.

The four ways coverage misleads

It rewards execution over assertion. As above, a line runs and counts, whether or not anything was checked. Teams incentivized on a coverage target will, rationally, write assertion-free tests to move the number. The metric corrupts the behavior it was meant to encourage — a textbook case of Goodhart's law: when a measure becomes a target, it stops being a good measure.

It treats all lines as equally important. A logging statement and the tax-calculation branch each count as one line. A team can hit 90% by covering trivial getters and boilerplate while leaving the revenue-critical logic in the uncovered 10%. Coverage has no concept of what matters.

It says nothing about inputs. Covering a function once means one path through it was exercised — usually the happy path. The bug lives in the empty cart, the expired token, the Unicode name, the concurrent request. Line coverage of that function can be 100% while every one of those cases is untested.

It is blind to what the code doesn't do. Coverage can only measure code that exists. The most dangerous defects are often missing behavior — an unhandled error state, a permission check that was never written. There is no line to cover, so coverage reports perfect health over a hole in the product.

Metrics that track quality more honestly

If coverage is a poor proxy, what is better? No single number replaces judgment, but these are far more correlated with real quality.

Mutation testing

Mutation testing deliberately introduces small bugs — flipping a > to >=, replacing a + with a -, removing a line — and then runs your suite. If a mutation survives (tests still pass with the bug in place), your tests didn't actually check that behavior. The mutation score measures how many injected defects your suite catches. Unlike coverage, it cannot be gamed by assertion-free tests, because a test that asserts nothing catches nothing. It directly answers the question coverage dodges: would these tests notice if the code were wrong?

Escaped-defect rate

The most honest quality metric is empirical: how many defects reach production per release, and how severe? Tracking escaped defects — and the flows they land in — tells you where your verification is genuinely thin, regardless of what any coverage report claims. A rising escaped-defect rate against flat coverage is proof the coverage number is fiction.

Critical-flow coverage

Not "what percentage of lines," but "which of the flows that actually matter are verified end-to-end." Can a user sign up, log in, add to cart, pay, and get a receipt — and is each of those protected? Ten well-chosen end-to-end scenarios over the flows that drive revenue are worth more than thousands of shallow unit tests over utility code.

Coverage still has a place

None of this means coverage is useless. Used correctly, it is a diagnostic, not a goal. A sudden drop in coverage on a pull request is a useful signal — new code arrived without tests. A file stuck at 0% coverage is worth a look. Coverage is good at pointing to what you definitely haven't tested. It is simply incapable of confirming that what you did test is correct. The failure is not the metric; it is treating a floor as a ceiling.

The pathology begins the moment a coverage percentage becomes a target in a dashboard or a merge gate. At that point it stops measuring quality and starts manufacturing the appearance of it.

Where BuniOD fits

The reason coverage endures despite its flaws is that the alternative — verifying that real behavior is correct across every flow that matters — has always been too expensive to maintain by hand. That is the constraint that changes when the work is automated.

BuniOD measures quality the way it should be measured: by the behavior of the product, not the lines of the code. It maps the flows that actually exist, ranks them by business impact, and generates end-to-end validation over the ones that matter — so the question is "does checkout still work?" rather than "did this line run?" Every result is validated before it is reported, which means the signal you get is about correctness, not execution. You can read about the security model or how it fits QA teams.

Conclusion

Code coverage measures execution, not correctness, and the gap between the two is where real defects live. It rewards assertion-free tests, treats trivial and critical code alike, ignores the inputs that break software, and is structurally blind to missing behavior. Kept as a diagnostic, it usefully flags untested code. Elevated to a target, it manufactures false confidence.

The metrics worth watching — mutation score, escaped-defect rate, and coverage of the flows that actually matter — all share the property coverage lacks: they measure whether your tests would notice if the product were broken. That is the only question a quality metric should answer, and it is the one a coverage percentage never has.