Key takeaways
- Observability means being able to understand a system's internal behaviour from its outputs: metrics, logs and traces. For testers, it turns a black box into a glass box, during tests and in production.
- Use telemetry as an oracle: assert that a request produced the right trace, made the expected downstream calls once, emitted no errors and stayed within its latency budget.
- In production, SLOs, canary analysis, synthetic checks and alert tests verify behaviour continuously. Production usage data tells you what your tests are missing.
Where this picks up
Shift-right testing and chaos engineering both depend on seeing what the system does. This article treats that visibility as a testing capability in its own right.
The building blocks
| Signal | What it shows | Testing use |
|---|---|---|
| Metrics | Numbers over time: request rate, error rate, latency percentiles, queue depth | SLO checks, canary comparison, performance budgets |
| Logs | Discrete events with context | Detect unexpected errors and warnings during tests |
| Traces | The path of one request across services, with timing per step | Assert on call graphs, retries, duplicates and latency per hop |
| Profiles | Where CPU and memory are spent | Find performance regressions |
OpenTelemetry has become the common open standard for producing this telemetry across languages and vendors.
Google's Site Reliability Engineering book (2016) suggests four golden signals for user-facing systems: latency, traffic, errors and saturation. They make a good default set for test oracles too.
Telemetry as a test oracle
Many important behaviours aren't visible in a response:
- Did the order service call the payment service exactly once, even with a retry?
- Did the cache get hit, or did every request go to the database?
- Was a warning logged that indicates silent data truncation?
- Did a background job actually run after the API returned 202?
With tracing in place, tests can assert on spans: this request produced one payment span, no error spans, a database span under 50 ms, and a message published to the right topic. This is sometimes called trace-based testing, and tools exist that let you write assertions directly against traces.
Testing in production with observability
SLOs and error budgets
A service level objective (SLO) states a target, for example "99.9% of checkout requests succeed within 1 s over 30 days". The error budget is the allowed shortfall. It gives testing and releases a shared, measurable definition of "good enough". When the budget burns fast, releases slow down and reliability work takes priority.
Automated canary analysis
Release a new version to a small canary group and statistically compare its metrics, such as errors, latency and business events, against the current version. Netflix and Google open-sourced a tool for this, Kayenta, in 2018. It's a test whose oracle is "no significant degradation compared with the baseline".
Synthetic monitoring
Scripted checks of critical journeys running against production every few minutes. Essentially automated end-to-end tests with an alert as the result.
Testing the alerts
An alert that never fires might be broken. Inject faults (see chaos engineering) and verify that the right alert fires, reaches the right person and links to a useful runbook.
Using production to improve tests
- Usage-based prioritisation: which endpoints, flows and devices do real users rely on most? Make sure they're the best tested.
- Untested paths: compare production traces with test traces. Code paths that are used in production but never exercised by tests are risks.
- Real data shapes: production telemetry reveals payload sizes, unusual characters and edge cases that should become test data, anonymised.
- Escaped defects: every incident's traces show exactly which path failed. Turn it into a regression test.
A team's checkout API passed all functional tests. Customers occasionally received two order confirmation emails, but nobody could reproduce it.
After adding distributed tracing, a tester compared traces of slow requests. Under load, the API gateway's timeout was shorter than the order service's, so the gateway retried while the first attempt was still running. Both attempts completed. The response the customer saw was correct; the trace showed two order-creation spans and two email spans.
The team added an idempotency key, aligned the timeouts and wrote a test that asserted, through the trace, exactly one order span per checkout, including under injected latency.
Lesson: the bug was invisible in responses and obvious in traces. Telemetry made it observable, and then testable.
Building observability into testing
- Instrument early: treat telemetry as a feature, reviewed and tested like any other.
- Collect telemetry in test environments too, not only in production.
- Add trace and log assertions to integration and end-to-end tests for critical flows.
- Define SLOs with product owners for key journeys, and use them as release criteria.
- Compare canaries automatically before full rollout.
- Close the loop: production gaps become tests; incidents become regression tests and alerts.
Do
Assert on what must happen inside the system ("exactly one charge per order; no error logs; downstream latency within budget"), using traces and logs from the test run.
Don't
Treat observability as an operations concern only. Without it, testers can only check what the UI or API chooses to show, and the most expensive bugs often hide underneath.
Check yourself
An API returns 202 Accepted and processes the request asynchronously. Functional tests check only the 202. How can observability strengthen these tests?
Show answer
Use the trace (or correlated logs) of the request to assert that the background work actually happened: the message was published, consumed exactly once, the database was updated and no error spans or logs appeared, within an agreed time. Add a metric-based check that the queue drains, and an alert test for messages landing in the dead-letter queue.
