Software Testing

Expert path · lesson 4 of 21

ExpertTest design6 min readPublished November 12, 2026

Test oracles: how do you know the result is correct?

Running a test is easy; deciding whether its output is right is the hard part. That's the oracle problem. Here are the kinds of oracles testers use, from specifications to differential and heuristic oracles, and how each can fool you.

On this page

Key takeaways

  • A test oracle is the mechanism that decides whether an observed behaviour is correct. Without a good oracle, more tests just produce more unexamined output.
  • Oracles come in families: specified (requirements, assertions, contracts), derived (previous versions, other implementations, relations between runs), implicit (crashes, hangs, memory errors) and human/heuristic.
  • Every oracle is partial and fallible. Know what each can and can't detect, and combine them.

Where this picks up

Every test case has an "expected result" field, and good test cases fill it with precise values. But for many systems, nobody can write the expected value down: search rankings, simulations, machine learning models, complex calculations, rendering. Earl Barr, Mark Harman and colleagues called this the oracle problem in a 2015 survey, and showed it's one of the central obstacles to automating testing.

Families of oracles

FamilyHow it decidesExamplesBlind spots
SpecifiedCompares with a stated expectationExpected values in test cases, assertions, design-by-contract pre/postconditions, schemasOnly as good as the specification; misses what nobody specified
Derived: regression / golden masterCompares with a previous version's outputSnapshot tests, approval testingApproves old bugs; noisy when output changes legitimately
Derived: differentialCompares with another implementation of the same thingTwo JSON parsers, the old and new pricing engine, a slow reference implementationBoth implementations can share the same bug
Derived: metamorphicChecks relations between outputs of related inputsSorting a list twice gives the same result; adding an irrelevant filter doesn't change totalsOnly detects violations of the chosen relations
ImplicitDetects universally bad behaviourCrashes, exceptions, timeouts, memory errors (sanitizers), 500 errors, assertion failuresSays nothing about wrong-but-plausible results
Human / heuristicA person judges with knowledge and heuristicsExploratory testing, reviews, usability sessionsCostly, inconsistent, subject to bias

Specified oracles: make expectations executable

The strongest everyday oracles live inside the code and tests:

  • Assertions and invariants: "balance is never negative", "total equals the sum of line items".
  • Contracts: preconditions and postconditions on functions and APIs.
  • Schemas: API responses validated against OpenAPI or JSON Schema.

They turn every test, including random and generated ones, into a check against meaningful rules, not just "didn't crash".

Derived oracles: when you can't state the answer

Regression and approval oracles

Record the output of a trusted version and compare future runs with it. That's very effective for detecting change, but it can't tell good change from bad. A golden master approved with a bug in it will defend that bug for years.

Differential testing

Run the same inputs through two implementations and compare. This is common for compilers, parsers, databases and migrations, where the old system acts as a reference for the new one.

A related idea is the pseudo-oracle: an independently written, simpler version of the computation, used only to check the real one. It can be slow; it only has to be right.

Partial and statistical oracles

Sometimes you can't know the exact answer but you can bound it: a route can't be shorter than the straight-line distance; probabilities sum to 1; a predicted price should fall within a plausible range. Partial oracles catch gross errors cheaply.

Implicit oracles: free, and underused

Crashes, unhandled exceptions, timeouts, memory corruption and HTTP 500s are always wrong. Tools like address and undefined-behaviour sanitizers turn silent memory errors into loud failures. Fuzzing campaigns rely almost entirely on implicit oracles.

Make sure they're on in your test runs: fail tests on unexpected log errors, enable sanitizers in test builds, and treat any 5xx as a failure in API tests.

Heuristic oracles: how humans judge

Experienced testers recognise problems through inconsistency. Michael Bolton and James Bach summarise these heuristics with the mnemonic FEW HICCUPPS. A product should be consistent with its History, its Image, Comparable products, Claims made about it, User expectations, the Product itself, its Purpose and Statutes and standards. It should also be consistent with Familiar problems, Explainability and the World.

These don't give a pass/fail. They give a reason to suspect a problem, which you then investigate.

Real-world example · Reinhart–Rogoff spreadsheet, 2013An independent recomputation found the error

A 2010 paper by economists Carmen Reinhart and Kenneth Rogoff reported that countries with public debt above 90% of GDP had sharply lower growth. It was widely cited in debates about austerity.

In 2013, graduate student Thomas Herndon tried to reproduce the results for a class assignment, with Michael Ash and Robert Pollin. Working from the original spreadsheet, they found that a formula's range left out five countries, along with other methodological choices they disputed. With their corrections, the average growth for high-debt countries came out positive, not negative. Reinhart and Rogoff acknowledged the spreadsheet error, while defending other parts of their analysis.

Lesson: the original results had no independent oracle. A second, independent computation, acting as a pseudo-oracle, found an error that years of citation hadn't.

Choosing and combining oracles

  1. For each test area, ask: how would we know it's wrong? If the answer is "we'd look at it", you have only a human oracle. Can you automate part of it?
  2. Turn on every implicit oracle you can: sanitizers, log-error checks, status-code checks.
  3. Add invariants and contracts to the code for rules that must always hold.
  4. Add a derived oracle where exact answers are unknowable: differential, metamorphic or partial.
  5. Review approved snapshots as carefully as code. Approving one is a test design decision.
  6. Measure oracle strength, for example with mutation testing: tests that execute code without checking its output strongly will let mutants survive.

Do

When migrating a pricing engine, run old and new side by side on real orders (a differential oracle), and investigate every difference until it's explained.

Don't

Count a test as passing because it ran without errors. Without an oracle that checks the result, a test exercises code but verifies nothing.

Check yourself

Your team tests a route-planning service. Nobody can say what the optimal route is for arbitrary inputs. Name three oracles you could still use.

Show answer

For example: implicit (no crashes or timeouts; valid JSON); partial (route length ≥ straight-line distance; route starts and ends at the requested points; every segment is a real road); metamorphic (reversing start and end on symmetric roads gives the same length; adding an unrelated closed road doesn't change the result); differential (compare with the previous version or a slower exact algorithm on small maps).