Key takeaways
- Testing every combination of parameters quickly becomes impossible: 4 parameters with 3 values each already make 81 combinations.
- Pairwise testing guarantees that every pair of values appears in at least one test, usually with a small fraction of the tests.
- It works because most failures are triggered by one or two interacting factors. For high-risk systems, step up to 3-way combinations.
Where this picks up
In Equivalence partitioning and boundary values: beyond the basics you met each choice coverage: every value appears at least once. That's cheap, but it can't find a bug that only happens when two specific values meet, such as Safari and PayPal.
Testing all combinations finds those, but it doesn't scale. Pairwise testing sits in between.
The explosion
A checkout page varies by:
| Parameter | Values |
|---|---|
| Browser | Chrome, Firefox, Safari |
| Payment | Card, PayPal, Bank transfer |
| Language | English, Romanian |
| User | Guest, Member |
All combinations: 3 × 3 × 2 × 2 = 36 tests. Add a device type and a currency and it's hundreds. Real systems have dozens of parameters.
Why pairs are usually enough
Researchers at the US National Institute of Standards and Technology (NIST) studied failures in real software to see how many factors had to combine to trigger each one.
In a 2004 paper, Richard Kuhn, Dolores Wallace and Albert Gallo analysed failure data from several real software systems. They asked: for each failure, how many parameters had to take specific values at the same time?
Most failures were triggered by one or two parameters. Fewer needed three, fewer still needed four, and in their data none needed more than six.
Lesson: you don't need every combination to find most bugs. Covering every pair catches the large majority; covering every triple catches more. That's the basis of pairwise and combinatorial testing.
A pairwise test set
For the checkout, these 9 tests cover every pair of values: every browser with every payment method, every language with every user type and so on:
| # | Browser | Payment | Language | User |
|---|---|---|---|---|
| 1 | Chrome | Card | English | Guest |
| 2 | Chrome | PayPal | Romanian | Member |
| 3 | Chrome | Bank transfer | English | Member |
| 4 | Firefox | Card | Romanian | Guest |
| 5 | Firefox | PayPal | English | Guest |
| 6 | Firefox | Bank transfer | Romanian | Member |
| 7 | Safari | Card | English | Member |
| 8 | Safari | PayPal | Romanian | Guest |
| 9 | Safari | Bank transfer | English | Guest |
Nine tests instead of 36. Check any pair, for example Safari + PayPal, Romanian + Guest, Bank transfer + Member, and you'll find it in at least one row.
Use a tool
Building pairwise sets by hand works for small examples. For real ones, use a generator:
- PICT, a free, open-source command-line tool from Microsoft.
- ACTS, NIST's free combinatorial testing tool, which also supports higher strengths like 3-way.
- Many test management tools have a pairwise feature built in.
You list the parameters and values; the tool generates the rows.
Constraints: impossible combinations
Real parameters have rules. Safari on Windows doesn't exist. Maybe bank transfer isn't offered to guests.
If you ignore constraints, the generator produces rows you can't run. Then you delete them, and lose the pairs they were covering. Instead, declare constraints in the tool, so it builds a valid set that still covers every possible pair.
When pairs aren't enough
For large models, constraints and measuring the coverage of existing suites, see Combinatorial testing at scale.
- Higher strength. For safety-critical, financial or highly configurable systems, use 3-way (all triples) on the riskiest parameters. Tests grow, but stay far below all combinations.
- Known interactions. If a rule explicitly combines three conditions, use a decision table for that rule. Pairwise is for parameters that shouldn't interact, to check that they really don't.
- Values still need choosing. Pairwise picks combinations of the values you give it. Use partitions and boundaries to choose good values first.
Do
Use pairwise for configuration testing: browsers, devices, settings, feature flags, locales. Add a few hand-picked combinations for the most common real-world setups.
Don't
Use pairwise as an excuse to skip analysis. If two parameters are meant to interact, model that rule directly instead of hoping a pair covers it.
Check yourself
Two parameters have 4 values each, and three parameters have 2 values each. How many tests would all combinations need, and what's the lowest possible number for pairwise?
Show answer
All combinations: 4 × 4 × 2 × 2 × 2 = 128 tests. Pairwise needs at least 4 × 4 = 16 (the product of the two largest parameters), and a good tool will usually get close to that.
