Key takeaways
- Any test suite samples a vanishingly small part of the possible inputs × states × environments × timings. The question is never "did we test everything?" but "did we sample where it matters, densely enough?"
- Simple statistics put hard limits on what testing can show: with zero failures in N independent random tests, you can only claim a failure rate below roughly 3/N with 95% confidence.
- "Enough" is a decision: the point where the expected value of more testing is lower than its cost. Make it explicit with risk, coverage, defect trends and confidence, not with gut feeling or the calendar.
Where this picks up
You know risk-based testing and the classic stopping signals in a test plan: exit criteria, coverage, open bugs. This article goes underneath them: what can a finite set of tests actually tell you, and how do you decide rationally when to stop?
The sample space is absurdly large
A function with two 32-bit integer inputs has 2⁶⁴ input pairs. At a billion tests per second, exhaustive testing would take more than 500 years. A real system multiplies that by internal state, configuration, concurrency, data in the database, versions of dependencies and time.
Edsger Dijkstra put the consequence plainly in 1970: testing can show the presence of bugs, but never their absence. Every test strategy is a strategy for choosing a sample.
Ways to sample
| Strategy | Sampling analogy | Strong at | Weak at |
|---|---|---|---|
| Partition and boundary analysis | Stratified sampling | Systematic coverage of specified behaviour | Faults the specification doesn't hint at |
| Risk-based selection | Weighted sampling | Spending effort where failure costs most | Risks nobody identified |
| Operational profile (random tests weighted by real usage) | Representative sampling | Estimating reliability as users experience it | Rare but critical inputs |
| Uniform random / fuzzing | Simple random sampling | Breadth, surprise, cheap at volume | Tiny failure regions with structure |
| Exploratory and experience-based | Adaptive, judgement-driven | Following anomalies, learning fast | Repeatability, measurable coverage |
Good strategies mix them. Systematic techniques find the faults the specification suggests. Random and exploratory techniques find the ones nobody imagined.
What random tests can and can't prove
Suppose a failure occurs with probability p on each independent test drawn from the operational profile. The chance that N tests miss it entirely is (1 − p)ᴺ.
The practical consequence is the rule of three: if you run N independent tests with zero failures, the 95% upper confidence bound on the failure probability is approximately 3/N.
| Tests without failure | You can claim (95%) a failure rate below about |
|---|---|
| 300 | 1 in 100 |
| 3,000 | 1 in 1,000 |
| 3,000,000 | 1 in 1,000,000 |
To claim one in a billion, you'd need around three billion representative tests without a single failure. In 1991, Ricky Butler and George Finelli used this argument to show that demonstrating the reliability required of some life-critical software, around 10⁻⁹ failures per hour, by testing alone would take an infeasible amount of test time. That's why such systems also rely on design analysis, formal methods and architectural safeguards.
A rare failure region
In 1994, mathematician Thomas Nicely noticed wrong results in number-theory computations on a new Intel Pentium. The cause was in the chip's floating-point division unit: a lookup table used by the division algorithm had five entries missing out of more than a thousand.
Only certain combinations of operands reached those entries, so the error was rare in ordinary use and wasn't caught before release. After public pressure, Intel offered replacement chips and took a charge of $475 million.
Lesson: the failure region was tiny and highly structured. Sampling that mimics typical use is very unlikely to hit it. What finds such regions is reasoning about the design itself, such as systematically checking every table entry, or sampling aimed at the structure.
Signals that you're approaching "enough"
No single signal is enough. Combine several:
- Risk coverage: every high and medium risk has a planned test response, executed. Residual risks are written down and accepted by an owner.
- Structural and model coverage: code, requirement, state or combinatorial coverage, with gaps explained rather than hidden.
- Defect discovery trend: new defects per unit of test effort are falling for the same test intensity. A falling curve because you stopped testing hard means nothing.
- Defect severity profile: late-found defects are minor, not serious.
- Independent estimates: with capture–recapture, two testers or methods test the same area independently. If A finds n₁ defects, B finds n₂ and m are found by both, a rough estimate of the total is n₁ × n₂ ÷ m. Many defects found by only one method suggests many remain.
- Confidence from operation: for systems with telemetry, failure rates from canaries and beta users are real operational samples.
Making the decision
- State what "enough" protects against: the top risks and their cost if they happen in production.
- Show the evidence for each: what was sampled, how, and what was found.
- Name the gaps: inputs, states or environments not sampled, and why.
- Estimate the value of more testing: which risks would more testing reduce, by how much, at what cost and delay?
- Let the risk owner decide, with the residual risk stated in business terms, and record the decision.
Do
"We stop here: all high risks are tested; defect discovery has flattened over the last 40 hours of exploratory sessions at constant intensity; payment edge cases were sampled with 2,000 generated cases and no failures. Residual risk: untested older Android versions, about 2% of users, accepted by the product owner."
Don't
"We ran 1,200 test cases and 98% passed, so we're done." That's a count of activity, not evidence about remaining risk.
Check yourself
A team ran 3,000 independent tests drawn from realistic usage and saw no failures. They claim 'the failure rate is below 1 in 100,000'. Is that justified?
Show answer
No. By the rule of three, 3,000 failure-free tests support a 95% upper bound of about 3/3,000 = 1 in 1,000, a hundred times weaker than claimed. To claim 1 in 100,000, they'd need roughly 300,000 representative failure-free tests, or other evidence like design analysis and operational data.
