Software Testing

Expert path · lesson 16 of 21

ExpertReliability5 min readPublished November 12, 2026

Testing event-driven and asynchronous systems

In event-driven systems, requests return before the work is done, messages arrive twice or out of order and services agree only eventually. Here's how to test delivery guarantees, idempotency, ordering, schema evolution and eventual consistency without flaky sleeps.

On this page

Key takeaways

  • Most brokers guarantee at-least-once delivery, so consumers will see duplicates. Ordering is usually guaranteed only within a partition or key. Design tests around these real guarantees, not the happy path.
  • Test the hard cases on purpose: duplicates, reordering, poison messages, consumer crashes mid-processing, broker outages and schema changes.
  • Replace sleep() with "eventually" assertions: poll until a condition is true or a timeout expires. Assert on end states and invariants, not on timing.

Where this picks up

You've seen distributed failure modes and how observability can show what happens after an API returns. Event-driven architectures make asynchrony the default: services communicate by publishing and consuming events through brokers like Kafka, RabbitMQ or cloud queues.

The guarantees you actually get

ConcernTypical realityConsequence for testing
DeliveryAt-least-once: messages may be redelivered after failures or rebalancesEvery consumer must be idempotent; test duplicates explicitly
OrderingPer partition or key only; across keys, no orderTest events for the same entity arriving out of order if keys can differ, and across partitions
ConsistencyEventual: read models and other services catch up laterTests must wait for conditions, and the product must handle stale reads
"Exactly-once"Usually exactly-once processing within specific boundaries, not end-to-end deliveryDon't assume it across systems; test side effects such as emails and payments for duplicates

What to test

Idempotency

Deliver the same event twice, or ten times, with the same ID. The outcome must be the same as once: one order, one email, one payment, one loyalty bonus. Also test redelivery after partial processing: the consumer updated the database, then crashed before acknowledging.

Ordering

For the same entity, send "address changed" and "order shipped" in both orders. Does the system handle the late one correctly, using version numbers, timestamps or state guards, or does it overwrite newer data with older?

Poison messages and retries

A malformed or unprocessable message must not block the queue forever. Test that it's retried a limited number of times with backoff, then moved to a dead-letter queue, with an alert, and that later messages keep flowing.

Failure during processing

Kill the consumer mid-message. Take the broker down and bring it back. Make a downstream dependency fail. Then check: no lost messages, no duplicate side effects, processing resumes.

The transactional outbox

When a service must update its database and publish an event, doing both separately risks one without the other. The common outbox pattern writes the event to a table in the same transaction, and a relay publishes it. Test crashes between write and publish, and verify the relay delivers exactly the events committed, eventually.

Schema evolution

Producers and consumers deploy independently, so events with old and new schemas coexist. Use a schema registry with compatibility rules, or message contracts, and test that consumers handle both old and new versions, and unknown fields.

Long-running workflows (sagas)

Multi-step business processes across services, such as reserve stock, charge card and ship, use compensating actions when a step fails. Test failure at every step and verify the compensations leave the system consistent.

Replacing sleeps with "eventually"

def eventually(condition, timeout=10.0, interval=0.1):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        if condition():
            return
        time.sleep(interval)
    raise AssertionError("condition not met within timeout")

publish(OrderPlaced(order_id="o-1"))
eventually(lambda: read_model.get("o-1").status == "CONFIRMED")
eventually(lambda: emails_sent_to("ana@example.com") == 1)

The test passes as soon as the condition is true, and fails clearly if it never is. Then add a "stays true" check where it matters, for example that the email count is still 1 a few seconds later, to catch late duplicates.

A layered approach

LayerWhat it testsHow
Handler unit testsLogic for each event, idempotency rulesCall the handler directly with events, including duplicates and old versions
Message contractsProducer and consumer agree on event shapesContract or schema-compatibility tests in each pipeline
Integration with a real brokerSerialization, acknowledgements, retries, dead-letter routingA broker in a container during tests
Fault injectionCrashes, broker outages, slow consumersKill and restart processes; pause the broker; inject latency
End-to-end flowsA few critical journeys across services"Eventually" assertions plus invariants
Real-world exampleDouble points on a busy weekend

A retailer's loyalty service consumed "order paid" events and added points. All tests passed: one event, one set of points.

During a weekend sale, the broker rebalanced consumer partitions several times under load. Messages that had been processed but not yet acknowledged were redelivered, and thousands of customers received double points.

The team made the handler idempotent by recording processed event IDs in the same transaction as the points update. They added tests that delivered each event twice, killed the consumer between processing and acknowledgement and checked the invariant "points equal the sum of distinct paid orders".

Lesson: at-least-once delivery means duplicates are not an edge case; they're guaranteed eventually. Tests have to deliver them on purpose.

Do

Assert invariants such as "each event's side effect happens once" while delivering duplicates, reordering events and restarting consumers mid-stream.

Don't

Publish one event, sleep for five seconds and check the database. It's slow, flaky and blind to duplicates, ordering and crash recovery.

Check yourself

A read model is updated asynchronously from events. A UI test sometimes fails because the new order isn't visible yet. What's the right fix in the test, and what question should you ask about the product?

Show answer

In the test, replace any fixed wait with an "eventually" assertion that polls until the order appears or a timeout expires. For the product, ask: what does the user see during the delay? If a user can place an order and then not find it, that's a usability issue that may need read-your-own-writes behaviour or a clear "processing" state, and that's worth testing too.