One node runs a causal chain of writes while NTP steps the clock back 100 ms mid-run. Plain physical timestamps go non-monotonic — event 5 commits at 940 ms though its cause committed at 1030 ms — so sorting by timestamp puts effects before their cause. A Hybrid Logical Clock's max-and-increment rule keeps the same chain strictly monotonic: (1030,c0) < (1030,c1) < (1030,c2) < (1030,c3).
One node performs a causal chain of writes — event k reads x=k-1 and writes x=k, so each event provably happens-after the last. Timestamp the chain two ways: plain physical (wall-clock) time, and a Hybrid Logical Clock (HLC). Partway through, NTP decides the clock is fast and steps it back 100 ms. The plain physical timestamps then go non-monotonic: event 5 commits at 940 ms though its own cause, event 4, committed at 1030 ms — a later event with a smaller timestamp. Sorting the events by physical time places e5, e6, e7 before e4, the write they read — causality inverted — and a snapshot read in the gap sees the effects but not the committed cause. The HLC applies one rule per event — pt = max(pt_last, phys); counter += 1 if pt unchanged else 0 — and its timestamps stay strictly monotonic ((1030,c0) < (1030,c1) < …), preserving the causal order while staying within the jump's bound of physical time. "Our clocks are NTP-synced, so timestamps are ordered" is wrong the instant the clock steps back; the HLC's max-and-increment rule buys the ordering back with no special hardware.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 10 — Consistency and Consensus, §"Hybrid logical clocks":
A hybrid logical clock combines a physical timestamp with a logical counter, giving it the best of both worlds: it stays close to the actual time of day, while also preserving the causal ordering of events like a Lamport clock.
The book presents the HLC (used by CockroachDB) as the clock that keeps physical time meaningful and never lets it regress. Here you watch a real backward step break the plain wall-clock ordering, then watch the HLC absorb the same step into its counter and keep the causal order intact.
The whole result turns on one line: pt = max(seen, physical); bump the counter on a tie or regression. These place why; one line each, with a free resource, and the one that carries the result flagged.
(pt, counter) pair: pt is physical, counter is logical.clock.step(-100) mid-chain.e5 (940) < e4 (1030) inverting the sort.pt = max(pt_last, phys); if pt did not advance (a tie, or physical time regressed) counter += 1, else counter = 0; receive first takes the max with the remote HLC. pt never decreases and the counter breaks ties, so the (pt, counter) pair is strictly monotonic. It shows up as e5 becoming (1030, c1) instead of going back to 940.ntpd docs on the step/slew threshold; the Ch. 9 exercise monotonic-vs-wallclock for what a backward step does to a duration, which this exercise's ordering failure is the sibling of. If only one is new, make it #4 — max-and-increment is the entire reason the HLC stays monotonic while tracking physical time.
An HLC is physical time that is never allowed to go backward. It carries a logical counter alongside the physical reading, and that counter does two jobs: it breaks ties when two events read the same physical millisecond, and it absorbs a backward jump — when physical time tries to regress, the HLC holds pt where it was and bumps the counter instead. So the HLC still tracks real time (its pt is a real millisecond value, within a bound of the wall clock) yet stays monotonic, because the pair (pt, counter) — compared lexicographically — can only ever increase.
(pt, counter). On each event pt = max(pt_last, phys); if physical time carried pt forward the counter resets to 0, otherwise it increments. receive(remote) takes the max with the remote HLC first, so a message's causality crosses the wire. The pair never decreases.Both stamp the identical causal chain; only the second keeps the order honest across the step. The whole thing is one script, code/hybrid_logical_clock.py.
cd study/ddia/ch10/code
python3 hybrid_logical_clock.pyDo not read the output yet — make each prediction first.
Predict. Seven causal writes, physical clock starting at 1000 ms and normally advancing +10 ms per event — but NTP steps the clock back 100 ms between event 4 and event 5. What physical timestamp does event 5 get, and how does it compare to event 4's?
940 < 1030 — a later event with a smaller timestamp. The clock ran forward to 1030, NTP stepped it back to ~930, and event 5 read 940. Event 5 causally happens-after event 4, but its physical timestamp is earlier. The timestamps are no longer a valid order.
Predict. A timestamp-ordered log (or a snapshot) sorts these events by their physical timestamp. What order comes out — and where do e5, e6, e7 land relative to e4, the write they read?
The sort puts effects before their cause. e5, e6, e7 — each of which read a value written by e4 — sort ahead of e4. Worse, a snapshot read at T = 1025 sees e5, e6, e7 (their ts ≤ 1025) but not e4 (ts 1030 > T): it exposes x=7 while the write that produced the chain's middle is invisible. Nothing errored; the ordering is just wrong.
Predict. Same seven events, same backward step, but each stamped by the HLC rule pt = max(pt_last, phys); counter += 1 if pt unchanged else 0. What HLC does event 5 get — does it go back to 940, or something else? Are the HLC timestamps strictly increasing?
Event 5 is (1030, c1), not 940. When physical time tried to regress, max held pt at 1030 and the tie-branch bumped the counter to 1. The next two events keep climbing on the counter — (1030,c1) < (1030,c2) < (1030,c3) — so the HLC timestamps are strictly increasing and sorting by them reproduces the true causal order e1..e7.
Predict. A peer node B, whose clock is ahead, sends a message stamped hlc = (1200, c4). Our node (whose HLC is (1030, c3), physical clock 960 ms) applies receive(). What does our HLC become — and is it strictly greater than the message's own stamp?
Our HLC jumps to (1200, c5). receive takes max(pt_last, remote_pt, phys) = 1200, and because that came from the remote, sets counter = remote_counter + 1 = 5. The receipt is now strictly greater than the send's (1200, c4), so the message's causality is preserved even though our own physical clock never read as high as 1200. This is how the HLC carries happened-before across a wire, exactly like a Lamport clock.
Predict. After absorbing a 100 ms backward step, how far can the HLC's pt run ahead of the true wall clock? Bounded, or does it drift without limit?
70 ms — bounded by the jump. At e7 the wall clock reads 960 ms but the HLC's pt is still 1030 ms: 70 ms ahead, the part of the 100 ms step that real time has not yet caught back up to. The counter, not an ever-growing physical offset, carried the ordering across the gap, so the HLC never drifts unboundedly from physical time.
e5 = 940 ms < e4 = 1030 ms, a later event with a smaller timestamp.e5 → e6 → e7 → e1 → e2 → e3 → e4 — effects before their cause — and a snapshot at T = 1025 shows e5, e6, e7 but not e4.(1000,c0) < (1010,c0) < (1020,c0) < (1030,c0) < (1030,c1) < (1030,c2) < (1030,c3); sorting by them reproduces the causal order.receive() from a peer at (1200, c4) lifts our HLC to (1200, c5) — strictly after the send.From first principles, plain physical time has an external authority — NTP — that is allowed to move it backward. That is not a bug in NTP; it is its job: when the local clock is too far from UTC to slew away gently, it steps it, possibly back. But an ordering source must be monotonic along causality — a later event may never receive a smaller value — and a quantity someone else can decrement has forfeited that property. So the moment the step lands, two events on either side of it are stamped from different calibrations of the same dial; comparing them is meaningless, and when the step is backward and larger than the real elapsed time, the later event's timestamp comes out smaller. That is exactly e5 = 940 < e4 = 1030, and it is why sorting inverts causality and the snapshot loses a committed write.
The HLC fixes this by clamping: on every event it sets pt = max(pt_last, phys). The max is the whole trick — it makes pt a running maximum of every physical reading ever seen, and a running maximum can never decrease. When physical time moves forward, pt follows it and the counter resets to 0 (a fresh millisecond, no tie to break). When physical time regresses (or two events share a millisecond), pt stays put — max ignores the smaller value — and the counter += 1. So a backward step doesn't invert anything; it just spends counter increments. Because (pt, counter) is compared lexicographically and pt is monotonic with the counter breaking every tie, the pair is strictly increasing by construction — a total order that never regresses. And since pt is still a real max of physical readings, it stays close to the wall clock: at most the un-recovered part of the last backward step ahead of it (70 ms here). receive extends the same max to remote timestamps, so happened-before crosses messages too — the HLC is a Lamport clock whose "counter" is anchored to real time.
pt staying "close" to real time depends on the clocks being roughly synced; a wildly wrong clock still produces a large-but-monotonic pt. The HLC guarantees a monotonic order, not an accurate instant.
(t,c0) then (t,c1)) impose an order the physical clock could not.Sources: DDIA 2e, Ch. 10, §"Hybrid logical clocks", §"Sequence number ordering" · Kulkarni et al., "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases" (2014).