Key takeaways
- Most brokers guarantee at-least-once delivery, so consumers will see duplicates. Ordering is usually guaranteed only within a partition or key. Design tests around these real guarantees, not the happy path.
- Test the hard cases on purpose: duplicates, reordering, poison messages, consumer crashes mid-processing, broker outages and schema changes.
- Replace
sleep()with "eventually" assertions: poll until a condition is true or a timeout expires. Assert on end states and invariants, not on timing.
Where this picks up
You've seen distributed failure modes and how observability can show what happens after an API returns. Event-driven architectures make asynchrony the default: services communicate by publishing and consuming events through brokers like Kafka, RabbitMQ or cloud queues.
The guarantees you actually get
| Concern | Typical reality | Consequence for testing |
|---|---|---|
| Delivery | At-least-once: messages may be redelivered after failures or rebalances | Every consumer must be idempotent; test duplicates explicitly |
| Ordering | Per partition or key only; across keys, no order | Test events for the same entity arriving out of order if keys can differ, and across partitions |
| Consistency | Eventual: read models and other services catch up later | Tests must wait for conditions, and the product must handle stale reads |
| "Exactly-once" | Usually exactly-once processing within specific boundaries, not end-to-end delivery | Don't assume it across systems; test side effects such as emails and payments for duplicates |
What to test
Idempotency
Deliver the same event twice, or ten times, with the same ID. The outcome must be the same as once: one order, one email, one payment, one loyalty bonus. Also test redelivery after partial processing: the consumer updated the database, then crashed before acknowledging.
Ordering
For the same entity, send "address changed" and "order shipped" in both orders. Does the system handle the late one correctly, using version numbers, timestamps or state guards, or does it overwrite newer data with older?
Poison messages and retries
A malformed or unprocessable message must not block the queue forever. Test that it's retried a limited number of times with backoff, then moved to a dead-letter queue, with an alert, and that later messages keep flowing.
Failure during processing
Kill the consumer mid-message. Take the broker down and bring it back. Make a downstream dependency fail. Then check: no lost messages, no duplicate side effects, processing resumes.
The transactional outbox
When a service must update its database and publish an event, doing both separately risks one without the other. The common outbox pattern writes the event to a table in the same transaction, and a relay publishes it. Test crashes between write and publish, and verify the relay delivers exactly the events committed, eventually.
Schema evolution
Producers and consumers deploy independently, so events with old and new schemas coexist. Use a schema registry with compatibility rules, or message contracts, and test that consumers handle both old and new versions, and unknown fields.
Long-running workflows (sagas)
Multi-step business processes across services, such as reserve stock, charge card and ship, use compensating actions when a step fails. Test failure at every step and verify the compensations leave the system consistent.
Replacing sleeps with "eventually"
def eventually(condition, timeout=10.0, interval=0.1):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if condition():
return
time.sleep(interval)
raise AssertionError("condition not met within timeout")
publish(OrderPlaced(order_id="o-1"))
eventually(lambda: read_model.get("o-1").status == "CONFIRMED")
eventually(lambda: emails_sent_to("ana@example.com") == 1)
The test passes as soon as the condition is true, and fails clearly if it never is. Then add a "stays true" check where it matters, for example that the email count is still 1 a few seconds later, to catch late duplicates.
A layered approach
| Layer | What it tests | How |
|---|---|---|
| Handler unit tests | Logic for each event, idempotency rules | Call the handler directly with events, including duplicates and old versions |
| Message contracts | Producer and consumer agree on event shapes | Contract or schema-compatibility tests in each pipeline |
| Integration with a real broker | Serialization, acknowledgements, retries, dead-letter routing | A broker in a container during tests |
| Fault injection | Crashes, broker outages, slow consumers | Kill and restart processes; pause the broker; inject latency |
| End-to-end flows | A few critical journeys across services | "Eventually" assertions plus invariants |
A retailer's loyalty service consumed "order paid" events and added points. All tests passed: one event, one set of points.
During a weekend sale, the broker rebalanced consumer partitions several times under load. Messages that had been processed but not yet acknowledged were redelivered, and thousands of customers received double points.
The team made the handler idempotent by recording processed event IDs in the same transaction as the points update. They added tests that delivered each event twice, killed the consumer between processing and acknowledgement and checked the invariant "points equal the sum of distinct paid orders".
Lesson: at-least-once delivery means duplicates are not an edge case; they're guaranteed eventually. Tests have to deliver them on purpose.
Do
Assert invariants such as "each event's side effect happens once" while delivering duplicates, reordering events and restarting consumers mid-stream.
Don't
Publish one event, sleep for five seconds and check the database. It's slow, flaky and blind to duplicates, ordering and crash recovery.
Check yourself
A read model is updated asynchronously from events. A UI test sometimes fails because the new order isn't visible yet. What's the right fix in the test, and what question should you ask about the product?
Show answer
In the test, replace any fixed wait with an "eventually" assertion that polls until the order appears or a timeout expires. For the product, ask: what does the user see during the delay? If a user can place an order and then not find it, that's a usability issue that may need read-your-own-writes behaviour or a clear "processing" state, and that's worth testing too.
