Software Testing

Expert path · lesson 10 of 21

ExpertTest design5 min readPublished November 12, 2026

Combinatorial testing at scale

Pairwise is a start. Real systems have dozens of parameters, constraints between them and risks that need 3-way or higher coverage. Here's how to model large input spaces, choose strengths, handle constraints, measure the combinatorial coverage of existing tests and run it all in CI.

On this page

Key takeaways

  • Covering arrays guarantee every t-way combination of values appears at least once. Their size grows roughly with vᵗ × log k (values, strength, parameters), so even large systems stay tractable at t = 2–4.
  • At scale, the hard parts aren't the algorithms: they're input parameter modelling, constraints and choosing mixed strengths by risk.
  • You can also measure the t-way coverage of tests you already have, and fill only the gaps, instead of replacing suites wholesale.

Where this picks up

The intermediate article on pairwise and combinatorial testing showed why pairs catch most interaction faults, with a 9-test example. Industrial systems are different: 30 configuration options, 15 feature flags, 8 browser/device families, 4 locales, user roles, data states and rules about which combinations are even possible.

Covering arrays and how they grow

A covering array of strength t for k parameters with v values each is a set of N tests in which every combination of values for every set of t parameters appears at least once.

The key insight: N grows logarithmically with the number of parameters and exponentially with strength. For example, 20 parameters with 3 values each have about 3.5 billion exhaustive combinations. Pairwise needs only around 15–20 tests; 3-way, several dozen; 4-way, a few hundred. Exact sizes depend on the generator.

That's why "how many parameters" is rarely the problem. "Which strength, for which parameters" is the real decision.

Mixed strength: put the depth where the risk is

Not all parameters deserve the same depth. Most generators support mixed-strength models:

  • 3- or 4-way among the parameters involved in pricing, permissions or safety.
  • 2-way across everything else.
  • Seeds: mandatory combinations you always include, such as the five most common real-world configurations from analytics, or combinations that failed before.

Research by NIST, including the 2004 study on fault interactions, suggests that the strength needed to catch all failures in the systems studied was at most six. For most business software, 2-way with targeted 3-way is a strong, affordable baseline.

Input parameter modelling: the real work

The quality of combinatorial testing depends almost entirely on the input model:

  1. Identify parameters from requirements, configuration files, feature flags, APIs and environments, not just UI fields.
  2. Choose values with equivalence partitioning and boundary analysis. Every value should represent a class of behaviour.
  3. Abstract carefully. "Order size" might become empty / one item / many items / maximum, not every number.
  4. Write constraints explicitly: impossible or forbidden combinations.
  5. Review the model with developers and domain experts before generating anything.

NIST's own case study of combinatorial testing applied to its ACTS tool found that input space modelling can be a significant undertaking that needs to be carefully managed. That matches industrial experience.

Constraints at scale

Real models have many rules: "Safari only on macOS and iOS", "invoice payment only for business accounts", "feature B requires feature A".

  • Don't post-filter. Generating first and deleting invalid rows loses coverage, silently.
  • Declare constraints in the generator so it builds valid tests that still cover every valid t-way combination.
  • Model invalid combinations separately: forbidden combinations should be rejected gracefully. Test them as their own small set of negative tests.

Measuring the coverage you already have

You may already have hundreds of configuration tests. Instead of discarding them, measure their combinatorial coverage: what percentage of all valid 2-way and 3-way combinations do they exercise? NIST provides a free tool for combinatorial coverage measurement.

Then generate tests only for the missing combinations. This turns combinatorial testing from a big-bang replacement into an incremental improvement.

Sequences and time

Combinatorial ideas also apply to order: if a system has events A–F that can occur in any order, a sequence covering array ensures every t-way order of events (A before C before E, and so on) appears in some test. That's useful for device start-up, installation steps and asynchronous workflows.

Real-world example · Lockheed Martin and NISTAn industrial pilot with NIST

Lockheed Martin and NIST ran a two-year pilot under a cooperative research agreement to evaluate combinatorial testing in a real industrial setting with complex software requirements.

The company used NIST's ACTS tool, supplemented by other combinatorial tools, across eight pilot projects. As reported by NIST and in Lockheed Martin's public statements about the pilot, results included improvements in test coverage and potential cost savings, by up to around 50% in some projects. The company said it planned to extend the technique more widely.

Lesson: in an organisation with complex systems, combinatorial methods delivered measurable benefits, once teams invested in modelling inputs and constraints well.

Running it in CI

  • Generate deterministically with a fixed seed, so runs are reproducible and diffs between versions of the model are reviewable.
  • Rotate: if a full 3-way array is too big for every pull request, run 2-way on every change and the 3-way array nightly.
  • Map rows to environments: configuration rows can drive containers, feature flag sets or device cloud sessions automatically.
  • Track model changes: a new parameter or value should trigger regeneration and a coverage report, like any other code change.

Do

Spend most of the effort on the input model, its values, constraints and strengths, and review it with the people who know the system.

Don't

Feed every possible field and value into a generator unreviewed. You'll get a mathematically perfect covering array of a meaningless model.

Check yourself

Your existing 400 configuration tests were written by hand over years. Management wants combinatorial testing. What would you do first?

Show answer

Measure the existing suite's combinatorial coverage against a reviewed input model: what share of valid 2-way and 3-way combinations it already covers. Then generate tests only for the gaps, and remove tests that add no new combinations. That keeps proven tests, shows the value in numbers and avoids a risky big-bang rewrite.