Key takeaways
- CI (continuous integration) merges and tests every change often. CD (continuous delivery or deployment) keeps every tested change ready to release, or releases it automatically.
- Continuous testing puts automated checks at every stage of the pipeline, so each change gets fast feedback on its risk.
- A pipeline only checks what you ask it to. Include non-functional checks, keep it fast and trusted, and fix flaky tests, or people will learn to ignore it.
Where this picks up
You know why regression testing needs automation, and which layers tests belong in. CI/CD is where those tests actually run, on every change, many times a day.
The words
- Continuous integration (CI): developers merge small changes into the main branch often, at least daily. Each merge triggers an automatic build and tests.
- Continuous delivery: every change that passes the pipeline is ready to release. A person decides when.
- Continuous deployment: every change that passes the pipeline is released automatically.
- Continuous testing: running automated tests as part of the pipeline to get immediate feedback on business risk for every change.
What runs where
| Stage | Typical checks | Goal |
|---|---|---|
| Before commit (developer's machine) | Linting, formatting, fast unit tests | Catch mistakes in seconds |
| Pull request | Build, static analysis, unit, API and contract tests, security scan of dependencies | Block bad changes before merge |
| After merge | Integration tests, end-to-end smoke tests in a test environment | Check the pieces work together |
| Before release | Wider end-to-end, performance, accessibility checks | Confidence for this release |
| After release | Synthetic monitoring, canary metrics, automatic rollback | Catch what testing couldn't |
Quality gates
A quality gate is a rule that stops the pipeline when a condition isn't met. Useful gates:
- All tests pass. No exceptions, no "known failures" left in place.
- No new critical issues from static analysis or security scans.
- No dependencies with known critical vulnerabilities.
- Performance budget: key pages or endpoints not slower than an agreed limit.
- Accessibility checks pass on changed pages.
Keeping the pipeline trustworthy
Speed
If the pull request pipeline takes 50 minutes, developers stop waiting for it. Keep it fast:
- Run tests in parallel.
- Move slow suites to later stages.
- Run only the tests affected by a change, where your tools support it (test impact analysis).
- Push checks down the test pyramid.
Stability
A test that fails randomly is worse than no test: it trains people to re-run instead of read. Track flaky tests, quarantine them out of the blocking path, fix them quickly and delete them if they can't be fixed.
A green main branch
Agree on a rule: if the main branch is broken, fixing it is everyone's first priority, or the change is reverted. A pipeline that's red half the time can't tell you anything.
Environments and data
Tests need environments that look like production and data they control. Many teams create a temporary environment for each pull request and generate test data automatically, so tests don't depend on each other or on a shared, messy test server.
What testers do in CI/CD
- Decide which checks go where, based on risk and speed.
- Write and maintain automated tests, especially API and end-to-end journeys.
- Investigate failures and separate real bugs from test or environment problems.
- Watch the pipeline's health: duration, flaky rate, how often gates are overridden.
- Keep exploring. The pipeline covers known risks. Exploratory testing finds the ones nobody has written a test for yet.
On 2 July 2019, Cloudflare deployed a new rule for its web application firewall. The rule contained a regular expression that, on real traffic, caused enormous backtracking and pushed CPU usage to 100% on servers across its global network.
Cloudflare's tests had checked that the rule worked: it detected what it was meant to detect. They didn't check how much CPU it used. Firewall rules were also deployed globally at once, by design, so they could respond quickly to new attacks.
Cloudflare's services were down for about 27 minutes. In its public post-mortem, the company committed to adding performance profiling of all rules to its test suite, moving to a regular expression engine with run-time guarantees and rolling out rule changes in stages.
Lesson: a pipeline checks what you ask it to. Functional tests passed; the missing check was non-functional. And a staged rollout would have limited the damage while monitoring caught it.
Do
Add a performance or resource check to the pipeline for changes that run on every request, like rules, middleware or database queries, even if it's a simple time budget.
Don't
Re-run a failing pipeline until it goes green without looking at why it failed. That's how real failures get waved through.
Check yourself
Your pull request pipeline takes 50 minutes, so developers merge before it finishes. What would you do?
Show answer
Make the blocking stage fast: parallelise the tests, move slow end-to-end and performance suites to the after-merge or nightly stage, push checks down to unit and API level and use test impact analysis where possible. Aim for a pull request pipeline of a few minutes. Then enforce the rule that merging requires a green pipeline.
