Software Testing

Expert path · lesson 1 of 21

ExpertReliability4 min readPublished November 9, 2026

Flaky tests: root causes and a remediation strategy

A flaky test passes and fails without any code change. Left alone, flakiness destroys trust in the whole suite. Here's how to find the causes and fix them systematically.

On this page

Key takeaways

  • A flaky test gives different results for the same code. The cost isn't the failure, it's the lost trust.
  • Most flakiness comes from a handful of causes: timing, shared state, order dependence, external services, and environment.
  • Treat flakiness as a defect with an owner: detect, quarantine, fix and prevent.

Why flakiness is expensive

When a test fails at random, people learn to ignore failures. Then a real failure is ignored too.

Every flaky failure also costs a re-run, a context switch and some doubt. Across a large team, that adds up to hours per day.

Real-world example · GoogleFlakiness at scale

In 2016, Google's testing team reported that about 1.5% of all test runs gave a flaky result, and almost 16% of their tests showed some level of flakiness.

At that scale, flaky failures were a constant cost: engineers had to re-run tests and investigate failures that weren't real bugs. Google built tooling to detect flaky tests automatically and keep them from blocking releases.

Lesson: even well-resourced teams can't eliminate flakiness by willpower. It needs to be measured and managed like any other defect.

The usual suspects

Timing and async

The test checks a result before the system has produced it. Fixed sleeps (sleep(2)) make this worse: too short on a slow machine, wasteful on a fast one.

Fix: wait for a condition (the element is visible, the request finished), never for a fixed time.

Shared state

Tests share a database row, a file, a cache or a global variable. One test changes it and another one fails.

Fix: every test creates its own data and cleans up, or runs against an isolated copy.

Order dependence

Test B only passes if test A ran first. It works on your machine but not when the CI runs tests in parallel or a different order.

Fix: run tests in random order regularly to expose hidden dependencies.

External services

Real payment gateways, email providers or third-party APIs time out or rate-limit you.

Fix: use stubs or fakes at the boundary in most tests. Keep a small, separate set of contract tests against the real service.

Environment

Time zones, locale settings, screen size, CPU speed, or "today's date" all differ between machines.

Fix: pin these explicitly in tests: fixed clock, fixed locale, fixed viewport.

A remediation process

  1. Detect. Automatically re-run failed tests in CI and record those that pass on retry. A pass after a failure, with no code change, means the test is flaky.
  2. Quarantine. Move confirmed flaky tests out of the blocking suite so they stop interrupting work. Keep running them so you still get the data.
  3. Assign. Every quarantined test gets an owner and a deadline. A quarantine without an owner is a graveyard.
  4. Diagnose. Reproduce under stress: run the test 100 times, in parallel, in random order, on a slow machine. Check logs and timestamps for the cause.
  5. Fix the cause, not the symptom. A longer timeout is rarely the real fix.
  6. Prevent. Add the cause to code review checklists and lint rules (for example, ban fixed sleeps).

Do

Track a flakiness rate per test and per suite, and review the worst offenders every sprint.

Don't

Delete a flaky test without understanding it. It may be the only one exposing a real race condition in the product.

Check yourself

A UI test fails about 1 time in 20, always with 'element not found'. Adding sleep(5) before the click makes it pass. Is it fixed?

Show answer

No, it's hidden. The test now waits five seconds on every run and can still fail on a slower machine. Wait for the element to be ready instead, and check whether the product has a real loading issue that users also see.