A steady stream of one event per second flows past a consumer that suffers a 60-second outage. Bucket the events by processing time and the drained backlog reads as a 61 events/second spike (DDIA Figure 12-8); re-bucket the same events by event time and the spike vanishes into a flat 1/second. Same data, two clocks — a 61× phantom.
A steady stream — exactly one event per second for 120 seconds — flows past a consumer whose job is to graph the per-second event rate. The true rate is flat: 1/second, start to finish. But the consumer suffers a 60-second outage (a redeploy). While it is down the events pile up in the broker's backlog, and when it restarts all 60 buffered events drain at nearly the same instant. Bucket the events by processing time — when the consumer got to each one — and those 60 events land in a single per-second bucket alongside the one live event of that second: a 61 events/second bar, a 61× phantom spike (DDIA Figure 12-8), flanked by empty buckets where the outage swallowed the traffic. Re-bucket the same events by their embedded event-time timestamp — when they happened in the world — and the spike vanishes: a flat 1/second across every bucket. Nothing in the data spiked. The consumer's own schedule did.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 12 — Stream Processing, §"Reasoning About Time", subsection "Event time versus processing time" (Figure 12-8):
Confusing event time and processing time leads to bad data. [...] if there is any kind of queueing, [processing time] is unlikely to be an accurate estimate of the time at which the event actually occurred.
The book's Figure 12-8 shows a stream restarting after a pause and a "spike" of events appearing at the restart — an artifact of a metric bucketed by when the consumer processed each event, not by when it happened. Here you manufacture that exact figure and watch the spike appear and disappear depending only on which timestamp you count by.
The whole surprise is which clock you count by. These help; each line notes where it shows up in the run.
event_time) and when your consumer got to it (processing_time). It shows up as the two fields on each event, and as the two histograms that disagree. This is the one that carries the whole result.bucket_by(): the same events, bucketed by two different keys, give two graphs.processing_time = 90.event_time is stamped by the producer and never changes; processing_time depends entirely on your consumer's schedule (pauses, restarts, backpressure). It shows up as the spike existing in one histogram and not the other, from identical events.Every event carries two clocks, and a rate graph has to pick one.
A backlog is what makes the two diverge violently. While the consumer is down, events don't stop happening — they queue in the broker. When it restarts it drains the whole queue in a burst, so many distinct event-times get stamped with one processing-time. Count by processing time and that compression reads as a spike; count by event time and the events spread back out across the seconds they truly belong to. Same events, two keys — one script, code/processing_time_spike.py.
cd study/ddia/ch12
python3 code/processing_time_spike.pyDo not read the output yet — make each prediction first.
Predict. One event per second for 120 seconds, then a graph of the per-second event rate. If you hadn't been told about the outage, what would you expect the rate graph to look like — and what is the true underlying rate at every second?
The real traffic is a flat line at 1/second — nothing spikes, nothing drops. The only wrinkle is when the consumer reads each event, not when it happened.
Predict. Bucket the events by processing time (the second the consumer read each one) and graph per-second counts around the restart. The consumer was down for the 60 seconds [30, 90) and drains the backlog at the restart instant, t=90. What is the peak per-second rate the graph shows — and what does it show during the outage?
A 61 events/second spike where the truth was 1/second. The 60 events that happened during the outage all got processing_time = 90, and the one live event that genuinely occurred at t=90 lands in the same bucket — 60 + 1 = 61. Every second of the outage reads as 0, then it all detonates in one bar. This bar chart is DDIA Figure 12-8, and every event in it is a real event that really happened one per second.
Predict. Take the identical 120 events and bucket them by their embedded event-time timestamp instead. Same data, different key. What does the same window t=85..95 look like now — and where does the spike go?
Flat 1/second — the spike is gone. Bucketing by event time puts each event back in the second it actually happened, so the outage's 60 events spread across event-times 30..89 exactly as they occurred. The peak rate and the floor rate are both 1/second, everywhere in the stream. The spike was never in the data.
Predict. Same events, two clocks. State the two peak rates side by side: what does the processing-time graph claim the peak traffic was, and what was it really?
61× versus 1×, from one set of events. The only thing that changed between the two graphs was the timestamp field the bucketing keyed on. One measured the world; the other measured the consumer's outage-and-catch-up schedule and reported it as if it were traffic.
A per-second rate graph is a count of events per time bucket, and the bucket key decides which clock's seconds you are counting into. Event time is stamped by the producer when the event happens and is then immutable — bucketing by it measures a property of the world. Processing time is read off the consumer's clock at the moment it happens to dequeue the event, so bucketing by it measures a property of the consumer's schedule. As long as the consumer keeps up, the two clocks track each other and the graphs agree. The instant the consumer stops keeping up, they come apart, because event time keeps advancing in the broker's backlog while processing time is frozen until the consumer runs again.
The spike is that gap made visible. During the 60-second outage, 60 events are produced — one per event-time second, 30 through 89 — and every one of them sits in the backlog with no processing time yet. At the restart instant, t=90, the consumer drains the entire backlog in one burst, so all 60 get the same processing-time stamp, and the one live event that genuinely arrives at t=90 joins them: 60 + 1 = 61 events in a single processing-time second. That is the arithmetic of the 61× spike — it is exactly the length of the pause (plus the one live event of the restart second) collapsed into one bucket. Symmetrically, the 60 buckets during the outage read as 0, because their events were carried forward to t=90. Re-key the identical events by event time and each one returns to the second it was produced in; the count per second is 1 again, everywhere, because the producer really did emit one per second. No event was created or destroyed — only relabeled.
processing_time ≈ event_time for every event, the two histograms are identical, and bucketing by processing time is harmless. The phantom spike is manufactured only by lag: an outage, a redeploy, a GC pause, backpressure, a slow downstream call — anything that pauses the consumer and lets a backlog form, which then drains in a burst. So the failure mode isn't "processing time is always wrong"; it's "processing time silently becomes wrong exactly when your system is under stress" — which is precisely when you are staring at the dashboard, and precisely when a fake 61× spike is most likely to be mistaken for a real incident.
Sources: DDIA 2e, Ch. 12, §"Reasoning About Time", §"Event time versus processing time" (Figure 12-8) · Akidau et al., "The Dataflow Model" (VLDB 2015) and "Streaming 101 / 102".