Software Testing

Expert path · lesson 2 of 21

ExpertStrategy6 min readUpdated November 12, 2026

Advanced risk-based testing

Beyond the likelihood × impact grid: where risk matrices mislead, how FMEA, hazard analysis and threat modelling find risks a workshop misses, how to use data to estimate likelihood and why the assumptions behind a risk assessment need testing too.

On this page

Key takeaways

  • Simple risk matrices are useful for conversation but mathematically weak: ordinal scores multiplied together can rank risks wrongly. For important decisions, move towards ranges, expected loss and explicit assumptions.
  • Use structured analysis, such as FMEA, hazard analysis and threat modelling, to find risks that brainstorming misses, and data, such as churn, defect history and incidents, to estimate likelihood.
  • The most dangerous risks hide in the assumptions of the analysis itself. Treat key assumptions as test targets.

Where this picks up

The intermediate article on risk-based testing covers the essentials: product vs project risk, a risk register, mapping risks to test responses and reporting residual risk. This article is about doing it rigorously in complex or high-stakes systems.

The trouble with risk matrices

A 1–3 or 1–5 likelihood × impact grid is quick and gets people talking. But it has known weaknesses. Louis Anthony Cox analysed them in a 2008 paper, "What's Wrong with Risk Matrices?":

  • Ordinal numbers aren't quantities. "Impact 3" isn't three times "impact 1". Multiplying them creates a false sense of precision.
  • Range compression. Very different risks land in the same cell. A €10,000 loss and a €10 million loss can both be "high impact".
  • Ranking errors. Cox showed that matrices can assign higher ratings to quantitatively smaller risks.
  • Hidden judgement. Scores feel objective, but they're opinions, often anchored by whoever speaks first.

That doesn't mean "don't use matrices". It means: use them to sort and discuss, and for the risks that drive big decisions, go further.

Going further: quantify roughly

For top risks, estimate in real units, with ranges:

RiskProbability per releaseImpact if it happensExpected loss
Incorrect VAT on cross-border invoices5–15%€200k–€1M (corrections, penalties)Roughly €10k–€150k
Checkout outage on peak day1–3%€400k–€800k lost salesRoughly €4k–€24k

Even rough numbers change conversations. They show which test investment pays for itself, and they expose disagreements: when one person estimates 1% and another 30%, you've found a knowledge gap worth closing.

Finding risks systematically

Workshops find the risks people already worry about. Structured techniques find the rest.

FMEA: failure mode and effects analysis

For each component or step, list how it could fail, the effect, the cause and the existing controls. Classic FMEA scores severity, occurrence and detection and multiplies them into a "risk priority number". That number has the same ordinal problems as risk matrices, so use the analysis for its structure, not its score.

Hazard analysis and STPA

For safety-relevant systems, hazard analysis starts from unacceptable outcomes and works backwards. Nancy Leveson's STPA (System-Theoretic Process Analysis) models the system as control loops and asks how unsafe control actions could happen: commands given wrongly, too late, too early or not at all. It's particularly good at finding risks from interactions between components and humans, not just component failures.

Threat modelling

For security, techniques like STRIDE walk through each data flow and trust boundary asking about spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege. Each credible threat becomes a security risk with a test response.

Fault trees

Start from one top-level failure and break it into the combinations of events that could cause it, using AND/OR logic. Fault trees show single points of failure clearly: any branch that leads to the top event through ORs alone.

Estimating likelihood with data

Opinion is a poor estimator of where bugs are. Data is better:

  • Code churn. Research at Microsoft by Nachiappan Nagappan and Thomas Ball found that measures of relative code churn were good predictors of defect density. Recently and heavily changed code is riskier.
  • Defect history. Defects cluster. Modules with many past defects tend to have more.
  • Complexity and coupling. Large, highly connected modules break more and break others.
  • Production signals. Incidents, error rates and support tickets per feature.
  • Change type. Data migrations, concurrency changes and permission changes are consistently high-likelihood.

Combine them into a hot-spot map of the codebase, and review it against the team's intuition. Where they disagree, investigate.

Assumptions are risks too

Every risk analysis rests on assumptions: "operators will notice within seconds", "this sensor rarely fails", "the third-party API validates input". If an assumption is wrong, the whole analysis is wrong, and the system can fail in exactly the way it was "proven" safe from.

Real-world example · Boeing 737 MAX, 2018–2019When the assumptions behind the safety analysis failed

The Boeing 737 MAX included a flight control function called MCAS, which could push the aircraft's nose down automatically. In the original design, MCAS acted on data from a single angle-of-attack sensor.

Investigations and reviews, including the US House Transportation Committee's report, found that the safety assessment relied on assumptions, including that pilots would recognise and respond to an erroneous activation within a few seconds, using existing procedures.

Two crashes, Lion Air Flight 610 in October 2018 and Ethiopian Airlines Flight 302 in March 2019, killed 346 people. In both, MCAS was activated by erroneous sensor data. The aircraft was grounded worldwide for about 20 months. The updated software compares data from both angle-of-attack sensors.

Lesson: the risk classification depended on how a single sensor fails and how quickly humans respond under stress. Those assumptions were themselves critical risks, and needed to be tested as rigorously as the software.

Testing assumptions

  1. List the assumptions behind each high-severity risk rating: "detected by monitoring", "users won't do X", "the partner validates Y".
  2. Rate each assumption: how confident are we, and what happens if it's false?
  3. Design tests for the critical ones: fault injection for "the system detects it", user studies or simulators for "people respond in time", contract tests for "the partner validates".
  4. Re-rate the risk with the results, and record which assumptions remain untested.

Governance: risk appetite and acceptance

At scale, risk-based testing is an organisational process:

  • Risk appetite: the organisation states which risks it won't accept, for example "no known data-loss defects in production", and which it will.
  • Ownership: each high risk has a named owner who can accept residual risk.
  • Traceability: risk → test response → evidence → decision, so audits and post-mortems can follow the reasoning.
  • Feedback: every escaped defect is reviewed. Was the risk identified? Was the likelihood or impact misjudged? Was the test response wrong? Feed the answer back into the techniques above.

Do

For the few risks that drive release decisions, estimate in real units with ranges, list the key assumptions and test the ones that would change the decision.

Don't

Present a colour-coded matrix as if its scores were measurements. The precision is false, and it can hide the risk that matters most.

Check yourself

A risk register marks 'payment provider outage' as low risk because 'automatic failover to the backup provider handles it'. What would an expert tester do with that rating?

Show answer

Treat the failover as a critical assumption, not a fact. Ask whether it has ever been tested end to end, then test it: inject a provider outage in a test environment, or in production with safeguards, and measure whether failover happens, how fast and whether any payments are lost or duplicated. Re-rate the risk based on the evidence.