Software Testing

Expert path · lesson 15 of 21

ExpertReliability5 min readPublished November 12, 2026

Observability as a testing tool

Logs, metrics and traces aren't just for operations. They're test oracles you can assert on, a map of how the system is really used and the only way to verify behaviour that only appears in production. Here's how expert testers use observability.

On this page

Key takeaways

  • Observability means being able to understand a system's internal behaviour from its outputs: metrics, logs and traces. For testers, it turns a black box into a glass box, during tests and in production.
  • Use telemetry as an oracle: assert that a request produced the right trace, made the expected downstream calls once, emitted no errors and stayed within its latency budget.
  • In production, SLOs, canary analysis, synthetic checks and alert tests verify behaviour continuously. Production usage data tells you what your tests are missing.

Where this picks up

Shift-right testing and chaos engineering both depend on seeing what the system does. This article treats that visibility as a testing capability in its own right.

The building blocks

SignalWhat it showsTesting use
MetricsNumbers over time: request rate, error rate, latency percentiles, queue depthSLO checks, canary comparison, performance budgets
LogsDiscrete events with contextDetect unexpected errors and warnings during tests
TracesThe path of one request across services, with timing per stepAssert on call graphs, retries, duplicates and latency per hop
ProfilesWhere CPU and memory are spentFind performance regressions

OpenTelemetry has become the common open standard for producing this telemetry across languages and vendors.

Google's Site Reliability Engineering book (2016) suggests four golden signals for user-facing systems: latency, traffic, errors and saturation. They make a good default set for test oracles too.

Telemetry as a test oracle

Many important behaviours aren't visible in a response:

  • Did the order service call the payment service exactly once, even with a retry?
  • Did the cache get hit, or did every request go to the database?
  • Was a warning logged that indicates silent data truncation?
  • Did a background job actually run after the API returned 202?

With tracing in place, tests can assert on spans: this request produced one payment span, no error spans, a database span under 50 ms, and a message published to the right topic. This is sometimes called trace-based testing, and tools exist that let you write assertions directly against traces.

Testing in production with observability

SLOs and error budgets

A service level objective (SLO) states a target, for example "99.9% of checkout requests succeed within 1 s over 30 days". The error budget is the allowed shortfall. It gives testing and releases a shared, measurable definition of "good enough". When the budget burns fast, releases slow down and reliability work takes priority.

Automated canary analysis

Release a new version to a small canary group and statistically compare its metrics, such as errors, latency and business events, against the current version. Netflix and Google open-sourced a tool for this, Kayenta, in 2018. It's a test whose oracle is "no significant degradation compared with the baseline".

Synthetic monitoring

Scripted checks of critical journeys running against production every few minutes. Essentially automated end-to-end tests with an alert as the result.

Testing the alerts

An alert that never fires might be broken. Inject faults (see chaos engineering) and verify that the right alert fires, reaches the right person and links to a useful runbook.

Using production to improve tests

  • Usage-based prioritisation: which endpoints, flows and devices do real users rely on most? Make sure they're the best tested.
  • Untested paths: compare production traces with test traces. Code paths that are used in production but never exercised by tests are risks.
  • Real data shapes: production telemetry reveals payload sizes, unusual characters and edge cases that should become test data, anonymised.
  • Escaped defects: every incident's traces show exactly which path failed. Turn it into a regression test.
Real-world exampleThe double call that every test missed

A team's checkout API passed all functional tests. Customers occasionally received two order confirmation emails, but nobody could reproduce it.

After adding distributed tracing, a tester compared traces of slow requests. Under load, the API gateway's timeout was shorter than the order service's, so the gateway retried while the first attempt was still running. Both attempts completed. The response the customer saw was correct; the trace showed two order-creation spans and two email spans.

The team added an idempotency key, aligned the timeouts and wrote a test that asserted, through the trace, exactly one order span per checkout, including under injected latency.

Lesson: the bug was invisible in responses and obvious in traces. Telemetry made it observable, and then testable.

Building observability into testing

  1. Instrument early: treat telemetry as a feature, reviewed and tested like any other.
  2. Collect telemetry in test environments too, not only in production.
  3. Add trace and log assertions to integration and end-to-end tests for critical flows.
  4. Define SLOs with product owners for key journeys, and use them as release criteria.
  5. Compare canaries automatically before full rollout.
  6. Close the loop: production gaps become tests; incidents become regression tests and alerts.

Do

Assert on what must happen inside the system ("exactly one charge per order; no error logs; downstream latency within budget"), using traces and logs from the test run.

Don't

Treat observability as an operations concern only. Without it, testers can only check what the UI or API chooses to show, and the most expensive bugs often hide underneath.

Check yourself

An API returns 202 Accepted and processes the request asynchronously. Functional tests check only the 202. How can observability strengthen these tests?

Show answer

Use the trace (or correlated logs) of the request to assert that the background work actually happened: the message was published, consumed exactly once, the database was updated and no error spans or logs appeared, within an agreed time. Add a metric-based check that the queue drains, and an alert test for messages landing in the dead-letter queue.