Key takeaways
- In distributed systems, partial failure is normal: messages are lost, delayed, duplicated or reordered; nodes pause and clocks drift. Concurrent code adds interleavings you can't control by default.
- Ordinary tests run in friendly conditions and almost never hit the bad interleavings. You need tools that create the bad conditions on purpose: race detectors, stress and interleaving tools, fault injection, partition testing.
- Check invariants and consistency guarantees ("no acknowledged write is lost"), not individual outputs. For the hardest systems, deterministic simulation makes rare failures reproducible.
Where this picks up
You've seen flaky tests caused by timing and shared state, and advanced state-based testing of recovery paths. Distributed systems combine both, at scale: many components, each with its own state, communicating over an unreliable network.
How distributed systems fail
| Failure | What happens | Typical bug |
|---|---|---|
| Message loss | A request or response never arrives | Client retries; server processes the operation twice |
| Delay and reordering | Messages arrive late or in a different order | "Cancel" processed before "create" |
| Duplication | The same message is delivered twice | Double charge, double email |
| Network partition | Some nodes can't reach others | Two nodes both believe they're the leader; split-brain writes |
| Process pauses | Garbage collection or VM pause freezes a node | A node acts on an expired lease |
| Clock skew | Machines disagree on the time | Wrong ordering by timestamp; tokens "expire" early |
| Partial failure | One dependency is slow, not down | Timeouts cascade; thread pools exhaust |
How concurrency fails
- Race conditions: the result depends on the timing of two operations, like check-then-act with no lock.
- Deadlocks and livelocks: threads wait for each other forever, or keep retrying without progress.
- Atomicity violations: a multi-step update is seen half-done by another thread.
- Memory visibility: one thread's writes aren't visible to another without proper synchronisation.
Such bugs might appear once in a million runs on a developer laptop, and daily under production load.
Techniques, from cheap to deep
1. Dynamic race detectors
Tools like Go's built-in race detector, ThreadSanitizer (for C/C++ and others) and Java's concurrency stress harnesses detect unsynchronised access during test runs. Turn them on in CI for concurrent code; they find bugs your tests merely execute.
2. Stress and interleaving tests
Run the same concurrent operation thousands of times with many threads, randomised delays and contention. Some tools systematically explore different thread schedules instead of hoping the scheduler hits them.
3. Fault injection at the network layer
Put a proxy between components that can add latency, drop or duplicate packets, cut connections or slow bandwidth. Then check that timeouts, retries, idempotency and circuit breakers behave as designed.
4. Partition and consistency testing (Jepsen-style)
Kyle Kingsbury's Jepsen project tests distributed databases by running concurrent clients while injecting partitions, pauses and clock changes. It records every operation and its result, then checks the history against the consistency model the system claims, such as linearizability or serializability. Jepsen analyses have found violated guarantees in many well-known databases and queues. The method generalises: record histories, inject faults, check guarantees.
5. Deterministic simulation testing
The FoundationDB team built their database so the whole cluster could run inside a deterministic simulator: simulated network, disks and clocks, driven by a single random seed. The simulator injects failures far more aggressively than reality, and any failure can be replayed exactly from its seed. It's expensive to build in from the start, but it turns once-a-year production races into reproducible test failures.
6. Model checking the design
For protocols, check the design before the code, with tools like TLA+. See formal methods and model checking.
Oracles: invariants, not outputs
Because timing is non-deterministic, exact outputs vary. Check properties that must always hold:
- No acknowledged write is lost.
- Every order is charged at most once.
- At most one leader per term.
- Balances sum to the same total after any set of transfers.
- Every message is eventually processed or sent to a dead-letter queue.
On 14 August 2003, a blackout spread across parts of the north-eastern United States and Ontario, affecting about 50 million people.
The joint US–Canada task force found that, at the utility FirstEnergy, the control-room alarm system had stopped working without operators realising. As transmission lines tripped, no alarms appeared, so operators didn't react while the situation could still be contained.
A code audit after the blackout by GE, which supplied the energy management system, traced the alarm failure to a race condition in the alarm-processing software, triggered by a particular combination of events on that day. The audit reviewed around a million lines of code.
Lesson: the race condition needed a rare combination of timing and events, so ordinary testing hadn't triggered it. And because it failed silently, nobody knew the safety net was gone. Concurrency testing and monitoring of the monitoring itself are both part of testing critical systems.
A practical programme
- Write down the guarantees the system claims: delivery semantics, consistency, idempotency, ordering.
- Turn on race detectors for concurrent code in CI.
- Add fault-injection tests for each external dependency: slow, down, flaky, duplicating.
- Make operations idempotent and test them with retries and duplicates.
- Record and check histories in concurrency tests against invariants.
- Run longer randomised campaigns nightly, and keep every seed that fails.
- Test the alarms: inject failures and verify that monitoring notices.
Do
Assert invariants over the whole run ("no order charged twice; every accepted order persisted") while injecting delays, retries and duplicate messages.
Don't
Add sleep() calls until a concurrent test passes. You've hidden the race in the test; it's still in the product.
Check yourself
A payment service retries calls to the bank when a request times out. In production, a few customers were charged twice. What tests would have caught it?
Show answer
Fault-injection tests where the bank processes the payment but the response is lost or delayed, so the client times out and retries, combined with an invariant check: each payment intent is charged at most once. The fix is usually an idempotency key sent with every attempt; then test duplicates, concurrent retries and retries after a long delay.
