Software Testing

Expert path · lesson 20 of 21

ExpertStrategy6 min readPublished November 12, 2026

Test adequacy: when can we trust our test suite?

Coverage percentages are easy to measure and easy to misread. Test adequacy asks a harder question: is this suite good enough to catch the faults that matter? Here are the main adequacy criteria, what research says about them and how to build a trustworthy picture.

On this page

Key takeaways

  • An adequacy criterion defines when a test suite is "enough" for some purpose: structural (statement, branch, MC/DC), fault-based (mutation), specification-based (requirements, partitions, states, combinations) or usage-based.
  • Coverage tells you what tests executed, not what they checked. Research shows coverage is only a modest predictor of fault detection once suite size is accounted for. It's a good finder of gaps, a poor measure of strength.
  • Trust comes from several independent signals: requirement and risk coverage, structural coverage, mutation score, escaped-defect data, flakiness and the quality of oracles.

Where this picks up

How much testing is enough? is about stopping: deciding with risk and confidence when to release. This article is about judging the suite itself: given these tests, how much should we trust them, and what's missing?

A short history of adequacy

In 1975, John Goodenough and Susan Gerhart's paper "Toward a Theory of Test Data Selection" asked what makes a test-selection criterion reliable and valid, starting a long line of research. In 1997, a survey by Hong Zhu, Patrick Hall and John May organised the many criteria proposed since into families. The core insight has held up: every criterion measures something specific, and none on its own guarantees the absence of faults.

Families of criteria

FamilyExamplesTells youDoesn't tell you
Structural (control flow)Statement, branch, condition, MC/DC, path coverageWhich code was executed, and which decisions took which outcomesWhether results were checked; whether missing code (unimplemented requirements) matters
Structural (data flow)Definition–use pairsWhether each computed value was used in a testSame as above
Fault-basedMutation scoreWhether tests detect small deliberate faultsFaults unlike the mutation operators; it's costly
Specification-basedRequirement, partition, boundary, decision table, state, combinatorial coverageWhether specified behaviour was exercisedBehaviour nobody specified
Usage-basedOperational profile coverageWhether real-world usage is representedRare but critical cases
Risk-basedCoverage of identified risksWhether effort matched riskRisks nobody identified

MC/DC

Modified condition/decision coverage requires showing that each condition in a decision independently affects the outcome. For if (a && (b || c)), you need test pairs where changing only a, only b or only c flips the result. Standards for the most critical avionics software, such as DO-178C Level A, require MC/DC. It's demanding, and much stronger than branch coverage for complex conditions.

What research says about coverage

Coverage clearly helps find untested code. Whether higher coverage means better fault detection is more nuanced. In a 2014 study, Laura Inozemtseva and Reid Holmes generated many test suites for large Java programs and found that, once the size of a suite was controlled for, coverage had only a low to moderate correlation with how many faults the suite detected. Other studies, like the mutation study by Just and colleagues the same year, found mutation score tracked real fault detection better than statement coverage.

Practical reading:

  • Low coverage is a reliable warning: untested code is untested.
  • High coverage is a weak reassurance: it says the code ran, not that a wrong result would have been noticed.
  • Never make a single coverage number a target. Teams will write tests that execute code without checking it.

Signals that build real trust

  1. Requirement and risk coverage: every requirement and identified risk traces to tests that exercise it, with gaps explained.
  2. Structural coverage, used to find gaps: review uncovered code, and decide whether it needs tests, is dead or is low-risk.
  3. Oracle strength: mutation testing on critical modules; assertion review; no "tests" that only check for absence of exceptions.
  4. Model coverage where relevant: state transitions, decision table columns, t-way combinations.
  5. Escaped-defect analysis: for each production defect, which test should have caught it and why didn't it? Trends over time are the most honest adequacy measure there is.
  6. Suite health: flakiness, run time and quarantined tests. A suite people don't trust isn't adequate, whatever its coverage.
Real-world example · SQLite100% MC/DC, and still more testing

SQLite, one of the most widely deployed database engines, documents its testing in detail on its "How SQLite Is Tested" page. As of version 3.42.0 (2023), the library had about 156,000 lines of C, and the project had around 590 times as much test code and test scripts.

Its core has 100% branch coverage and 100% MC/DC under its TH3 test harness. On top of that, it uses mutation testing to check that every branch actually makes a difference to the tests, and runs fuzzing continuously, including through Google's OSS-Fuzz.

The page even notes a tension: code tested to 100% MC/DC tends to be more vulnerable to the kinds of problems fuzzing finds, so it needs both.

Lesson: one of the most thoroughly tested projects in the world doesn't trust any single adequacy criterion. It combines structural coverage, fault-based checks and randomised testing, because each finds what the others miss.

Making it practical

  • Set criteria by risk: MC/DC and high mutation scores for safety- or money-critical modules; branch coverage plus risk coverage elsewhere.
  • Report adequacy as a profile, not a number: "Payments: 100% risk coverage, 94% branch, mutation score 81%, no escaped defects in 6 months."
  • Review test code with the same care as production code; weak assertions and over-mocking are adequacy problems.
  • Revisit after incidents: an escaped defect is evidence about which criterion was missing.

Do

Use coverage reports to find uncovered risky code, and mutation testing to find unchecked behaviour, then fix the specific gaps.

Don't

Require "80% coverage" across the codebase as the definition of a good test suite. It rewards executing code, not verifying it.

Check yourself

Two services both report 90% branch coverage. Service A has a mutation score of 85% and no escaped defects this year; service B has 40% and five escaped defects. What does this tell you, and what would you do about B?

Show answer

Coverage alone made them look equal, but B's tests execute code without checking it well, which the low mutation score and escaped defects confirm. For B: review surviving mutants in critical modules, strengthen assertions and oracles, add tests for the paths behind the escaped defects and track mutation score and escapes over time, rather than coverage.