Key takeaways
- An adequacy criterion defines when a test suite is "enough" for some purpose: structural (statement, branch, MC/DC), fault-based (mutation), specification-based (requirements, partitions, states, combinations) or usage-based.
- Coverage tells you what tests executed, not what they checked. Research shows coverage is only a modest predictor of fault detection once suite size is accounted for. It's a good finder of gaps, a poor measure of strength.
- Trust comes from several independent signals: requirement and risk coverage, structural coverage, mutation score, escaped-defect data, flakiness and the quality of oracles.
Where this picks up
How much testing is enough? is about stopping: deciding with risk and confidence when to release. This article is about judging the suite itself: given these tests, how much should we trust them, and what's missing?
A short history of adequacy
In 1975, John Goodenough and Susan Gerhart's paper "Toward a Theory of Test Data Selection" asked what makes a test-selection criterion reliable and valid, starting a long line of research. In 1997, a survey by Hong Zhu, Patrick Hall and John May organised the many criteria proposed since into families. The core insight has held up: every criterion measures something specific, and none on its own guarantees the absence of faults.
Families of criteria
| Family | Examples | Tells you | Doesn't tell you |
|---|---|---|---|
| Structural (control flow) | Statement, branch, condition, MC/DC, path coverage | Which code was executed, and which decisions took which outcomes | Whether results were checked; whether missing code (unimplemented requirements) matters |
| Structural (data flow) | Definition–use pairs | Whether each computed value was used in a test | Same as above |
| Fault-based | Mutation score | Whether tests detect small deliberate faults | Faults unlike the mutation operators; it's costly |
| Specification-based | Requirement, partition, boundary, decision table, state, combinatorial coverage | Whether specified behaviour was exercised | Behaviour nobody specified |
| Usage-based | Operational profile coverage | Whether real-world usage is represented | Rare but critical cases |
| Risk-based | Coverage of identified risks | Whether effort matched risk | Risks nobody identified |
MC/DC
Modified condition/decision coverage requires showing that each condition in a decision independently affects the outcome. For if (a && (b || c)), you need test pairs where changing only a, only b or only c flips the result. Standards for the most critical avionics software, such as DO-178C Level A, require MC/DC. It's demanding, and much stronger than branch coverage for complex conditions.
What research says about coverage
Coverage clearly helps find untested code. Whether higher coverage means better fault detection is more nuanced. In a 2014 study, Laura Inozemtseva and Reid Holmes generated many test suites for large Java programs and found that, once the size of a suite was controlled for, coverage had only a low to moderate correlation with how many faults the suite detected. Other studies, like the mutation study by Just and colleagues the same year, found mutation score tracked real fault detection better than statement coverage.
Practical reading:
- Low coverage is a reliable warning: untested code is untested.
- High coverage is a weak reassurance: it says the code ran, not that a wrong result would have been noticed.
- Never make a single coverage number a target. Teams will write tests that execute code without checking it.
Signals that build real trust
- Requirement and risk coverage: every requirement and identified risk traces to tests that exercise it, with gaps explained.
- Structural coverage, used to find gaps: review uncovered code, and decide whether it needs tests, is dead or is low-risk.
- Oracle strength: mutation testing on critical modules; assertion review; no "tests" that only check for absence of exceptions.
- Model coverage where relevant: state transitions, decision table columns, t-way combinations.
- Escaped-defect analysis: for each production defect, which test should have caught it and why didn't it? Trends over time are the most honest adequacy measure there is.
- Suite health: flakiness, run time and quarantined tests. A suite people don't trust isn't adequate, whatever its coverage.
SQLite, one of the most widely deployed database engines, documents its testing in detail on its "How SQLite Is Tested" page. As of version 3.42.0 (2023), the library had about 156,000 lines of C, and the project had around 590 times as much test code and test scripts.
Its core has 100% branch coverage and 100% MC/DC under its TH3 test harness. On top of that, it uses mutation testing to check that every branch actually makes a difference to the tests, and runs fuzzing continuously, including through Google's OSS-Fuzz.
The page even notes a tension: code tested to 100% MC/DC tends to be more vulnerable to the kinds of problems fuzzing finds, so it needs both.
Lesson: one of the most thoroughly tested projects in the world doesn't trust any single adequacy criterion. It combines structural coverage, fault-based checks and randomised testing, because each finds what the others miss.
Making it practical
- Set criteria by risk: MC/DC and high mutation scores for safety- or money-critical modules; branch coverage plus risk coverage elsewhere.
- Report adequacy as a profile, not a number: "Payments: 100% risk coverage, 94% branch, mutation score 81%, no escaped defects in 6 months."
- Review test code with the same care as production code; weak assertions and over-mocking are adequacy problems.
- Revisit after incidents: an escaped defect is evidence about which criterion was missing.
Do
Use coverage reports to find uncovered risky code, and mutation testing to find unchecked behaviour, then fix the specific gaps.
Don't
Require "80% coverage" across the codebase as the definition of a good test suite. It rewards executing code, not verifying it.
Check yourself
Two services both report 90% branch coverage. Service A has a mutation score of 85% and no escaped defects this year; service B has 40% and five escaped defects. What does this tell you, and what would you do about B?
Show answer
Coverage alone made them look equal, but B's tests execute code without checking it well, which the low mutation score and escaped defects confirm. For B: review surviving mutants in critical modules, strengthen assertions and oracles, add tests for the paths behind the escaped defects and track mutation score and escapes over time, rather than coverage.
