Software Testing

Intermediate path · lesson 15 of 19

IntermediateAutomation5 min readPublished November 12, 2026

Continuous testing in CI/CD

In a CI/CD pipeline, every change is built, tested and ready to release within minutes. Continuous testing makes that safe: the right checks at each stage, quality gates that mean something and a pipeline people trust.

On this page

Key takeaways

  • CI (continuous integration) merges and tests every change often. CD (continuous delivery or deployment) keeps every tested change ready to release, or releases it automatically.
  • Continuous testing puts automated checks at every stage of the pipeline, so each change gets fast feedback on its risk.
  • A pipeline only checks what you ask it to. Include non-functional checks, keep it fast and trusted, and fix flaky tests, or people will learn to ignore it.

Where this picks up

You know why regression testing needs automation, and which layers tests belong in. CI/CD is where those tests actually run, on every change, many times a day.

The words

  • Continuous integration (CI): developers merge small changes into the main branch often, at least daily. Each merge triggers an automatic build and tests.
  • Continuous delivery: every change that passes the pipeline is ready to release. A person decides when.
  • Continuous deployment: every change that passes the pipeline is released automatically.
  • Continuous testing: running automated tests as part of the pipeline to get immediate feedback on business risk for every change.

What runs where

StageTypical checksGoal
Before commit (developer's machine)Linting, formatting, fast unit testsCatch mistakes in seconds
Pull requestBuild, static analysis, unit, API and contract tests, security scan of dependenciesBlock bad changes before merge
After mergeIntegration tests, end-to-end smoke tests in a test environmentCheck the pieces work together
Before releaseWider end-to-end, performance, accessibility checksConfidence for this release
After releaseSynthetic monitoring, canary metrics, automatic rollbackCatch what testing couldn't

Quality gates

A quality gate is a rule that stops the pipeline when a condition isn't met. Useful gates:

  • All tests pass. No exceptions, no "known failures" left in place.
  • No new critical issues from static analysis or security scans.
  • No dependencies with known critical vulnerabilities.
  • Performance budget: key pages or endpoints not slower than an agreed limit.
  • Accessibility checks pass on changed pages.

Keeping the pipeline trustworthy

Speed

If the pull request pipeline takes 50 minutes, developers stop waiting for it. Keep it fast:

  • Run tests in parallel.
  • Move slow suites to later stages.
  • Run only the tests affected by a change, where your tools support it (test impact analysis).
  • Push checks down the test pyramid.

Stability

A test that fails randomly is worse than no test: it trains people to re-run instead of read. Track flaky tests, quarantine them out of the blocking path, fix them quickly and delete them if they can't be fixed.

A green main branch

Agree on a rule: if the main branch is broken, fixing it is everyone's first priority, or the change is reverted. A pipeline that's red half the time can't tell you anything.

Environments and data

Tests need environments that look like production and data they control. Many teams create a temporary environment for each pull request and generate test data automatically, so tests don't depend on each other or on a shared, messy test server.

What testers do in CI/CD

  • Decide which checks go where, based on risk and speed.
  • Write and maintain automated tests, especially API and end-to-end journeys.
  • Investigate failures and separate real bugs from test or environment problems.
  • Watch the pipeline's health: duration, flaky rate, how often gates are overridden.
  • Keep exploring. The pipeline covers known risks. Exploratory testing finds the ones nobody has written a test for yet.
Real-world example · Cloudflare, 2019The tests passed; the CPU didn't

On 2 July 2019, Cloudflare deployed a new rule for its web application firewall. The rule contained a regular expression that, on real traffic, caused enormous backtracking and pushed CPU usage to 100% on servers across its global network.

Cloudflare's tests had checked that the rule worked: it detected what it was meant to detect. They didn't check how much CPU it used. Firewall rules were also deployed globally at once, by design, so they could respond quickly to new attacks.

Cloudflare's services were down for about 27 minutes. In its public post-mortem, the company committed to adding performance profiling of all rules to its test suite, moving to a regular expression engine with run-time guarantees and rolling out rule changes in stages.

Lesson: a pipeline checks what you ask it to. Functional tests passed; the missing check was non-functional. And a staged rollout would have limited the damage while monitoring caught it.

Do

Add a performance or resource check to the pipeline for changes that run on every request, like rules, middleware or database queries, even if it's a simple time budget.

Don't

Re-run a failing pipeline until it goes green without looking at why it failed. That's how real failures get waved through.

Check yourself

Your pull request pipeline takes 50 minutes, so developers merge before it finishes. What would you do?

Show answer

Make the blocking stage fast: parallelise the tests, move slow end-to-end and performance suites to the after-merge or nightly stage, push checks down to unit and API level and use test impact analysis where possible. Aim for a pull request pipeline of a few minutes. Then enforce the rule that merging requires a green pipeline.