Key takeaways
- A flaky test gives different results for the same code. The cost isn't the failure, it's the lost trust.
- Most flakiness comes from a handful of causes: timing, shared state, order dependence, external services, and environment.
- Treat flakiness as a defect with an owner: detect, quarantine, fix and prevent.
Why flakiness is expensive
When a test fails at random, people learn to ignore failures. Then a real failure is ignored too.
Every flaky failure also costs a re-run, a context switch and some doubt. Across a large team, that adds up to hours per day.
In 2016, Google's testing team reported that about 1.5% of all test runs gave a flaky result, and almost 16% of their tests showed some level of flakiness.
At that scale, flaky failures were a constant cost: engineers had to re-run tests and investigate failures that weren't real bugs. Google built tooling to detect flaky tests automatically and keep them from blocking releases.
Lesson: even well-resourced teams can't eliminate flakiness by willpower. It needs to be measured and managed like any other defect.
The usual suspects
Timing and async
The test checks a result before the system has produced it. Fixed sleeps (sleep(2)) make this worse: too short on a slow machine, wasteful on a fast one.
Fix: wait for a condition (the element is visible, the request finished), never for a fixed time.
Shared state
Tests share a database row, a file, a cache or a global variable. One test changes it and another one fails.
Fix: every test creates its own data and cleans up, or runs against an isolated copy.
Order dependence
Test B only passes if test A ran first. It works on your machine but not when the CI runs tests in parallel or a different order.
Fix: run tests in random order regularly to expose hidden dependencies.
External services
Real payment gateways, email providers or third-party APIs time out or rate-limit you.
Fix: use stubs or fakes at the boundary in most tests. Keep a small, separate set of contract tests against the real service.
Environment
Time zones, locale settings, screen size, CPU speed, or "today's date" all differ between machines.
Fix: pin these explicitly in tests: fixed clock, fixed locale, fixed viewport.
A remediation process
- Detect. Automatically re-run failed tests in CI and record those that pass on retry. A pass after a failure, with no code change, means the test is flaky.
- Quarantine. Move confirmed flaky tests out of the blocking suite so they stop interrupting work. Keep running them so you still get the data.
- Assign. Every quarantined test gets an owner and a deadline. A quarantine without an owner is a graveyard.
- Diagnose. Reproduce under stress: run the test 100 times, in parallel, in random order, on a slow machine. Check logs and timestamps for the cause.
- Fix the cause, not the symptom. A longer timeout is rarely the real fix.
- Prevent. Add the cause to code review checklists and lint rules (for example, ban fixed sleeps).
Do
Track a flakiness rate per test and per suite, and review the worst offenders every sprint.
Don't
Delete a flaky test without understanding it. It may be the only one exposing a real race condition in the product.
Check yourself
A UI test fails about 1 time in 20, always with 'element not found'. Adding sleep(5) before the click makes it pass. Is it fixed?
Show answer
No, it's hidden. The test now waits five seconds on every run and can still fail on a slower machine. Wait for the element to be ready instead, and check whether the product has a real loading issue that users also see.
