Software Testing

Expert path · lesson 13 of 21

ExpertReliability5 min readPublished November 12, 2026

Chaos engineering and failure testing

Systems fail in production whether you plan for it or not. Chaos engineering injects failures deliberately, as controlled experiments with a hypothesis, a small blast radius and an abort button, to find weaknesses before they cause outages.

On this page

Key takeaways

  • Chaos engineering is experimentation, not random breakage: define a measurable steady state, hypothesise that it holds under a specific failure, inject the failure and look for a difference.
  • Safety comes from minimising blast radius, having clear abort conditions and growing gradually, from staging to small slices of production.
  • It covers more than server crashes: latency, dependency failures, region loss, bad config, clock problems, and testing the recovery itself, including backups, failover and runbooks.

Where this picks up

Shift-right testing introduced chaos engineering as one way of learning from production. Testing distributed systems showed how partial failures break assumptions. Chaos engineering applies those ideas to the whole running system, continuously.

The principles

The community-written Principles of Chaos Engineering describe the approach:

  1. Build a hypothesis around steady-state behaviour. Use measurable outputs that matter to users, such as orders per minute, successful logins and p99 latency, not internal metrics like CPU.
  2. Vary real-world events. Server crashes, network latency, dependency errors, traffic spikes, expired certificates.
  3. Run experiments in production, where the real traffic, data and configuration are, when you're ready and with safeguards.
  4. Automate experiments to run continuously, so resilience doesn't decay as the system changes.
  5. Minimise blast radius. Start small, contain the impact and stop at the first sign of harm.

Designing an experiment

ElementExample
Steady stateCheckout success rate ≥ 99.5%, p99 latency < 800 ms
Hypothesis"If one of three recommendation service instances is terminated, checkout steady state holds"
FaultTerminate one instance; or add 2 s latency to calls to the recommendation service
Scope (blast radius)1% of traffic in one region, for 10 minutes
Control groupThe other 99% of traffic, for comparison
Abort conditionsCheckout success < 99% or error budget burn above threshold → stop immediately
ObservationDashboards and traces for checkout, recommendations and dependencies
OutcomeHypothesis held, or a weakness found with a ticket and owner

A failed hypothesis is a success for the experiment: you found a weakness in a controlled way instead of during an outage.

What to inject

  • Infrastructure: instance or container termination, disk full, CPU or memory pressure.
  • Network: latency, packet loss, DNS failure, partition between services.
  • Dependencies: a downstream API returning errors, slow responses or malformed data.
  • Scale: sudden load, cold caches, a thundering herd after recovery.
  • Platform: zone or region failure, expired certificates, clock skew.
  • People and process: on-call responds to an injected incident in a game day. Do alerts fire? Is the runbook right?
Real-world example · Netflix, 2011Designing for failure by causing it

When Netflix moved its streaming service to the cloud, it assumed individual servers could disappear at any time. To make sure services really tolerated that, it built Chaos Monkey, a tool that randomly terminates production instances.

In a July 2011 post, "The Netflix Simian Army", the company described a growing family of such tools. Chaos Monkey kills instances; Chaos Gorilla simulates the loss of an entire availability zone; others inject latency or check for configuration problems. Netflix later released Chaos Monkey as open source, and its engineers helped shape the wider discipline of chaos engineering.

Lesson: if failures are certain to happen, causing them regularly and safely turns resilience from a design intention into a tested property.

Getting started safely

  1. Make sure you can see: clear steady-state metrics, dashboards, tracing and alerts. Without observability, chaos is just damage.
  2. Start in staging with known failures, such as killing an instance or slowing a dependency.
  3. Run a game day: a scheduled, team-wide exercise with a facilitator and an abort plan.
  4. Move to production with a tiny blast radius, business hours and the whole team aware.
  5. Fix what you find before running bigger experiments.
  6. Automate the experiments that pass, so they keep running as the system changes.

Tools include Chaos Monkey, cloud provider services such as AWS Fault Injection Service and Azure Chaos Studio, and open-source platforms like LitmusChaos and Chaos Mesh for Kubernetes.

Do

Write the hypothesis and abort conditions before injecting anything, and announce production experiments to everyone who might be paged.

Don't

Kill random servers in production "to see what happens" without steady-state metrics or a way to stop. That's an outage you caused, not an experiment.

Check yourself

Your service has a fallback that should serve cached prices if the pricing service is down. How would you design a chaos experiment for it?

Show answer

Steady state: checkout success rate and the share of orders with a valid price. Hypothesis: if the pricing service returns errors for 1% of traffic, checkout steady state holds and those requests are served from the cache. Fault: inject errors or timeouts on pricing calls for that 1%. Abort if checkout success drops below the threshold. Observe fallback usage, cache freshness and errors. Then check prices served were within allowed staleness.