Software Testing

Expert path · lesson 8 of 21

ExpertTest design5 min readPublished November 12, 2026

Metamorphic testing: testing without a reliable oracle

When you can't know the correct output for a single input, you can often know how outputs should relate across inputs. Metamorphic testing checks those relations, and has found bugs in compilers, search engines, simulations and machine-learning systems.

On this page

Key takeaways

  • A metamorphic relation (MR) describes how the output should change when the input changes in a known way. For example, adding a filter to a search can only shrink the results.
  • Metamorphic testing runs a source input, transforms it into a follow-up input and checks that the two outputs satisfy the relation. No expected value is needed.
  • It's especially valuable where exact answers are unknowable: scientific software, compilers, search and ranking, simulations, ML models. The skill lies in finding strong relations that real faults would break.

Where this picks up

You've seen the oracle problem: for many programs, you can't say what the right output is. Metamorphic testing is one of the most practical answers. It was introduced by T.Y. Chen, S.C. Cheung and S.M. Yiu in a 1998 technical report, and has since become a major research and industrial technique.

The idea in one example

You can't easily check that sin(1.234) returns exactly the right value. But you know that sin(x) = sin(π − x). So:

  1. Source test: compute a = sin(1.234).
  2. Follow-up test: compute b = sin(π − 1.234).
  3. Check the relation: a ≈ b, within floating-point tolerance.

If the relation is violated, at least one of the outputs is wrong, even though you never knew either correct value.

Relations in practice

SystemMetamorphic relation
Search engineAdding an extra AND term returns a subset of the original results
E-commerce filterResults for "price < 100" contain all results for "price < 50"
Sorting / rankingShuffling the input order doesn't change the sorted output
Shortest pathThe shortest A→B route is no longer than A→C plus C→B
StatisticsMultiplying all values by k multiplies the mean by k; adding a constant doesn't change the variance
Image classifierA slight brightness change shouldn't change the predicted label
CompilerRemoving code that is never executed for a given input doesn't change the program's output for that input
SimulationDoubling the time step within stable limits gives approximately the same final state

Relations can be equalities (same output), inclusions (subset or superset), inequalities (no larger than) or approximate (within a tolerance).

Finding good relations

Good metamorphic relations are:

  • Grounded in the domain: they come from mathematics, physics, business rules or user expectations, not from the implementation.
  • Strong: a typical fault breaks them. "Output is not null" is a relation, but a weak one.
  • Diverse: several different relations exercise different parts of the logic.

A useful method: list the input transformations the system should be insensitive to (reordering, scaling, irrelevant additions, equivalent rewrites) or sensitive to in a predictable way (stricter filters, larger inputs). Each one suggests a relation.

Automating it

Metamorphic testing combines naturally with generated inputs:

@given(st.lists(product_strategy(), min_size=1))
def test_stricter_filter_returns_subset(products):
    store = load(products)
    broad = set(search(store, max_price=100))
    narrow = set(search(store, max_price=50))
    assert narrow <= broad          # metamorphic relation: subset

Each generated source input yields a follow-up input automatically, so you can run thousands of checks without a single hand-written expected value.

Real-world example · EMI compiler testing, PLDI 2014Over a hundred compiler bugs, no expected outputs needed

Testing a compiler is the oracle problem at its hardest: for a random program, what's the "correct" machine code?

Vu Le, Mehrdad Afshari and Zhendong Su introduced Equivalence Modulo Inputs (EMI). Run a program with a given input and record which code is never executed. Then create variants by deleting some of that dead code. For that input, every variant must produce the same output as the original. If a compiled variant gives a different result, the compiler is wrong.

In their 2014 PLDI paper, they reported 147 confirmed, unique bugs in GCC and LLVM found over eleven months, most of them miscompilations, the hardest kind of compiler bug to spot. Many were fixed by the compiler developers.

Lesson: one strong metamorphic relation, applied automatically to many programs, found bugs in two of the most heavily tested pieces of software in the world.

Limitations

  • Relations are partial oracles. A program can satisfy every chosen relation and still be wrong, for example if it's consistently wrong in a way the relations don't constrain.
  • Some relations need tolerances, especially with floating point and ML. Too tight produces false alarms; too loose hides bugs.
  • Debugging is indirect. A violation says one of two runs is wrong, not which one.

Mitigate by combining relations, using several follow-ups per source and pairing metamorphic tests with other oracles.

Do

Derive relations from domain knowledge with domain experts: "In our risk model, increasing exposure can never decrease the capital requirement."

Don't

Rely on one weak relation, like "the output has the same length". It will pass for most real bugs and give false confidence.

Check yourself

You're testing a product search API that ranks results by relevance. Suggest two metamorphic relations.

Show answer

For example: (1) adding a filter (category, maximum price) returns a subset of the unfiltered results; (2) changing the letter case of the query, or adding extra spaces, returns the same results in the same order. Others: adding a product that doesn't match the query doesn't change the results; requesting page 2 with page size 10 equals items 11–20 of a single page of 20.