study › DDIA › Chapter 12 · Stream Processing

Load Balancing Plus Redelivery Reorders an "In-Order" Stream

A broker sends m1…m5 in order and fans them across two consumers that ACK on completion. C2 stalls mid-m3; its lease expires and the broker redelivers m3 to C1 — which has already processed m4. Completion order comes out m1, m2, m4, m3, m5: m3 lands after m4. Run the same stall against a single partitioned log and the order is preserved — m1, m2, m3, m4, m5. It takes load balancing and redelivery together to reorder.

Concept

A broker sends five messages in order — m1, m2, m3, m4, m5 — and hands each to a consumer that ACKs on completion. Two features the broker offers for throughput and reliability — load balancing (fan the messages across several consumers) and redelivery (re-send a message whose lease expires because a consumer stalled) — combine to break the overall order. C2 stalls mid-m3; its lease expires and the broker redelivers m3 to C1 — but C1 has already processed m4. So m3 (sent 3rd) is ACKed after m4 (sent 4th): completion order comes out m1, m2, m4, m3, m5. Then run the same stall against a single partitioned log with one consumer: nothing overtakes m3, and completion order equals send order. Same fault, same messages; the only difference is whether a second consumer existed to redeliver to.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 12 — Stream Processing, §"Multiple consumers" / §"Acknowledgments and redelivery" (Figure 12-2). The claim under test, tightly paraphrased:

Combining load balancing with redelivery inevitably leads to messages being reordered: a message redelivered after a consumer fails to acknowledge it can be processed after messages that were originally sent later.

The book pairs load balancing (spreading work across consumers to scale throughput) with acknowledgments and redelivery (a message unacknowledged before its lease expires is sent to another consumer). Figure 12-2 shows the consequence: redelivering a stalled message lands it after a later one. Here you build the smallest real thing that exhibits it, then watch a partitioned log avoid it.

Prerequisites

The surprise needs two broker features working together; each line notes where it shows up.

Message brokers and acknowledgments — a broker holds messages and hands them out; the consumer sends an ACK when it finishes, and only then may the broker forget the message. It shows up as every ACKs mN [DONE] line.
Load balancing across consumers — to scale, the broker fans messages over several consumers (each message goes to one of them). It shows up as the fan-out C1 <- m1, m4, m5 | C2 <- m2, m3.
Redelivery on timeout / nack — if a consumer holds a message past its lease without ACKing (it crashed or stalled), the broker assumes failure and redelivers the message to another consumer. It shows up as the lease on m3 EXPIRED -> redeliver to C1 event.
Ordering guarantees: per-partition vs global — a log preserves order only within one partition (one consumer, one offset sequence); a shared queue fanned across consumers has no single sequence to preserve. It shows up as the contrast between Scenario A and Scenario B.

Where to learn the prerequisites Brokers, ACKs, redelivery (#1–#3): DDIA §"Message brokers compared to databases", §"Multiple consumers", and §"Acknowledgments and redelivery"; RabbitMQ's "Consumer Acknowledgements and Publisher Confirms" guide is a concrete free reference for leases and redelivery. Per-partition vs global order (#4): DDIA §"Partitioned logs" and the Apache Kafka docs §"Consumers" (a partition is consumed in order by exactly one consumer in a group). Load balancing (#2) and redelivery (#3) jointly carry the result — either one alone preserves order; it is the combination that reorders. That is the whole point, so if either is unfamiliar, start there.

Environment these numbers came from Deterministic, so the ordering reproduces exactly. Captured on macOS 26.5.2 (Darwin 25.5.0), arm64, Python 3.15.0a8, standard library only — no Docker, no deps. The dispatch, stall, and redelivery are modeled explicitly on a virtual clock, so the interleaving is fixed every run. Fixed timings: a healthy message takes 10 ticks; the lease is 8 ticks; a hung consumer stalls 30 ticks.

Mental model

A broker preserves order only within a single consumer's stream. Fanning messages across consumers for throughput already lets their work interleave; redelivering a failed message then reinserts it at a different point in the sequence, and a later message overtakes an earlier one.

Scenario A — shared queue, load balancing + redelivery. The broker prefetches m1, m4, m5 to C1 and m2, m3 to C2. C2 finishes m2, starts m3, then hangs (a GC pause). Its lease on m3 expires; the broker redelivers m3 to C1, which by then has already ACKed m4. C1 processes the redelivered m3 next, so m3 lands after m4.
Scenario B — single partitioned log, one consumer. One consumer owns the log and processes it in offset order. The same stall happens on m3, but there is no second consumer to redeliver to, so nothing overtakes m3: the consumer blocks on it (head-of-line blocking) and resumes. Order is preserved — slower, but ordered.

Both scenarios are one script, code/redelivery_reorder.py; the completion order is computed by the simulation, not hard-coded.

Setup

cd study/ddia/ch12/code
python3 redelivery_reorder.py

Do not read the output yet — make each prediction first.

Step 1 · the send order

Predict. The producer sends m1, m2, m3, m4, m5 in that order, and the broker "tries to preserve order." Before any consumer runs, what order do you expect the messages to be completed in?

What completion order does "in-order" suggest?

A broker sends five messages in order, expecting an ACK per message. send order: m1, m2, m3, m4, m5

The intuitive answer is "m1, m2, m3, m4, m5 — that's what in-order means." Hold that prediction against Scenario A.

Step 2 · load balancing plus redelivery

Predict. The broker fans out C1 ← m1, m4, m5 and C2 ← m2, m3. C2 stalls on m3 (a GC pause); its lease expires and m3 is redelivered to C1. In what order do the five ACKs actually land?

What order do the ACKs actually land in?

==================================================================== Scenario A: shared queue, load balancing across C1 + C2 + redelivery ==================================================================== Broker fans out (prefetch): C1 <- m1, m4, m5 | C2 <- m2, m3 Lease = 8 ticks; a healthy message takes 10; a hung consumer stalls 30. [t= 0] C1 starts m1 [t= 0] C2 starts m2 [t=10] C1 ACKs m1 [DONE] [t=10] C1 starts m4 [t=10] C2 ACKs m2 [DONE] [t=10] C2 starts m3, then HANGS (GC pause) -- no ACK coming [t=18] broker: lease on m3 (held by C2) EXPIRED -> redeliver to C1 [t=20] C1 ACKs m4 [DONE] [t=20] C1 starts m3 [t=30] C1 ACKs m3 [DONE] [t=30] C1 starts m5 [t=40] C2 wakes and finishes m3, but its lease is gone -> ACK rejected (duplicate) [t=40] C1 ACKs m5 [DONE] send order: m1, m2, m3, m4, m5 completion order: m1, m2, m4, m3, m5 -> m3 was sent BEFORE m4, but completed AFTER it -- delivery order was NOT preserved.

Completion order is m1, m2, m4, m3, m5 — m3 lands after m4. C1's effective processing order was m1, m4, then the redelivered m3, then m5. Nothing errored and no message was lost; the "in-order" stream simply came out reordered. (Notice too that at t=40 the hung C2 wakes and tries to ACK m3 — a duplicate the broker must reject, the other hazard of redelivery.)

Step 3 · the same stall on a partitioned log

Predict. Now one consumer owns a single partitioned log and processes it in offset order. The same GC pause hits m3. Does m3 still complete after m4?

Does m3 still overtake into the wrong place?

==================================================================== Scenario B: single partitioned log, one consumer, SAME stall on m3 ==================================================================== No load balancing: one consumer processes the log in offset order. [t= 0] C1 starts m1 [t=10] C1 ACKs m1 [DONE] [t=10] C1 starts m2 [t=20] C1 ACKs m2 [DONE] [t=20] C1 starts m3, then HANGS (GC pause, 30 ticks) [t=20] no second consumer exists -> broker cannot redeliver; nothing overtakes m3 [t=50] C1 ACKs m3 [DONE] [t=50] C1 starts m4 [t=60] C1 ACKs m4 [DONE] [t=60] C1 starts m5 [t=70] C1 ACKs m5 [DONE] send order: m1, m2, m3, m4, m5 completion order: m1, m2, m3, m4, m5 -> completion order == send order -- delivery order preserved.

Completion order is m1, m2, m3, m4, m5 — send order, preserved. The stall still happened; m3 blocked the log for 30 ticks (head-of-line blocking, so m4 and m5 finish later than in Scenario A). But with no competitor to redeliver to, nothing could overtake m3. Redelivery alone did not reorder — it took load balancing too.

What you should see

Send order is m1, m2, m3, m4, m5.
Scenario A (load balancing + redelivery) completes m1, m2, m4, m3, m5 — m3 lands after m4, and the woken C2 produces a rejected duplicate ACK of m3.
Scenario B (single partitioned log, same stall) completes m1, m2, m3, m4, m5 — order preserved, though m3's stall delays m4 and m5.

Why

Preserving order requires a single serial processor: some one place through which every message passes in sequence, so "before" and "after" are well defined. Load balancing deliberately gives that up. The moment the broker fans m1…m5 across two consumers to gain throughput, there is no single sequence any more — C1's stream and C2's stream advance independently, and their completions interleave by whoever finishes first. That interleaving alone can already scramble order; the broker's "tries to preserve order" only ever meant within one consumer's stream.

Redelivery then delivers the coup de grâce. When C2's lease on m3 expires at t=18, the broker does the safe thing — it assumes C2 failed and hands m3 to a live consumer so the message is not lost. But C1 is not at the same point in the sequence C2 was: C1 has already moved past m4. Reinserting m3 into C1's stream puts it after m4, and m3 (sent 3rd) is ACKed 4th. The redelivery is correct in isolation — it prevents message loss — and reordering is the price. That is why the two features are jointly incompatible with a global order: load balancing removes the single sequence, and redelivery reinserts a message at a point in a different sequence than where it started.

The boundary — the reorder needs shared-queue load balancing and redelivery Remove either and order returns. A single consumer (Scenario B) has one serial stream, so nothing can overtake a stalled message — it just blocks (head-of-line blocking), trading latency for order. A partitioned log gives each partition to exactly one consumer, which reads it in strict offset order and does not redeliver a slow message to a competitor; a stuck message blocks its partition instead of jumping into another. So ordered systems don't use a shared queue with redelivery — they partition the log and accept head-of-line blocking as the cost of order. Scenario B is slower precisely because it refuses to parallelize past the stall.

Go deeper

Kafka vs AMQP/JMS. Kafka's partition-per-consumer model (Scenario B) keeps order but caps parallelism at the partition count and suffers head-of-line blocking; AMQP/JMS shared queues (Scenario A) scale consumers freely but give up order on redelivery. Re-run with more messages and a second partition and watch order hold within each partition but not across them.
Why exactly-once pushes you to partitioned logs. The woken C2 at t=40 tried to re-ACK m3 — a duplicate. Combine that with the reorder and you see why getting both ordering and exactly-once effects steers systems toward partitioned logs with per-partition offsets (idempotent, ordered) rather than shared queues with acknowledgments (at-least-once, unordered).
Shrink the lease. Set LEASE below TICK and watch the broker redeliver messages that were merely slow, not failed — manufacturing duplicates and reorders out of healthy consumers. Lease tuning is a real operational knob.

Sources: DDIA 2e, Ch. 12, §"Multiple consumers", §"Acknowledgments and redelivery" (Figure 12-2), §"Partitioned logs" · Apache Kafka documentation, §"Consumers".