studyDDIAChapter 9 · The Trouble with Distributed Systems

Measuring a Duration With the Wall Clock Goes Negative; Monotonic Can't

A duration is elapsed = end - start and must be ≥ 0. Read start and end from a time-of-day clock and let NTP step it back 5 s during a 2 s measurement, and elapsed comes out −3.000 s — a lease check elapsed > 10 then reads False forever and the lease never expires. Time the same interval with a monotonic clock, which has no external authority for NTP to correct, and the step is invisible: elapsed stays the true +2.000 s.


Concept

A duration is the simplest measurement in the world: elapsed = end - start, and it must be >= 0. Read start and end from a time-of-day clock (System.currentTimeMillis(), time.time()) and the number is only meaningful if nothing moved the clock in between. But a time-of-day clock is a synchronized value — NTP can step it backward at any moment to correct drift. Step it backward by 5 s during a 2 s measurement and end - start comes out −3.000 s: a negative duration. A lease/timeout check built on it — if elapsed > 10: release — then reads −3 > 10 as False and the lease looks un-expired forever. Measure the same interval with a monotonic clock — a local counter that only ever counts up, with no external authority for NTP to correct — and the step is invisible: elapsed stays the true +2.000 s. This is the book's core reason there are two clocks.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 9 — The Trouble with Distributed Systems, §"Monotonic Versus Time-of-Day Clocks" and §"Relying on Synchronized Clocks":

A monotonic clock is suitable for measuring a duration (time interval), such as a timeout or a service's response time [...] The name comes from the fact that they are guaranteed to always move forward (whereas a time-of-day clock may jump back in time).

The book's rule is blunt: use a time-of-day clock to know what time it is, and a monotonic clock to know how long something took. Here you watch what happens when you break that rule — you compute a duration from the time-of-day clock and an NTP step corrupts it.

Prerequisites

The surprise is that subtraction — end - start — can produce a negative duration. These place why; one line each, with a free resource, and the one that carries the result flagged.

  1. Time-of-day vs monotonic clocks (carries the result) — a time-of-day clock reports civil time (a Unix timestamp) and can be reset by an authority; a monotonic clock reports elapsed ticks from an arbitrary origin and only counts up. It shows up as the two measure_* functions computing the same end - start with opposite guarantees.
  2. NTP slewing vs stepping — NTP corrects a wrong clock two ways: slewing (gradually speeding/slowing it so it converges) or stepping (jumping it instantly, used when the error is large). A step is what jumps a duration. It shows up as clock.step(-5.0).
  3. Leap seconds — occasionally a UTC minute has 61 (or, in principle, 59) seconds; systems have historically smeared or stepped over this, another way a time-of-day clock jumps under your feet. It shows up as a second real-world cause of the same backward jump the mock injects.
  4. Why durations need a monotonic source — a timeout, a benchmark, a lease, an RPC latency are all how long, not what time; each must be read from a clock no external authority can reset. It shows up as the lease check mis-firing on the wall-clock duration.
Where to learn the prerequisites The two clocks and why durations need monotonic (#1, #4): DDIA §"Monotonic Versus Time-of-Day Clocks"; Python time module docs on time.monotonic() vs time.time() — the docs state plainly that monotonic "cannot go backward." NTP slewing vs stepping and leap seconds (#2, #3): the ntpd documentation on the step/slew threshold; Google's "leap smear" write-up for how a leap second is spread out to avoid a backward step. If only one is new, make it #1 — every other line is a consequence of the two clocks answering two different questions.
Environment these numbers came from macOS 26.5.2 (Darwin 25.5.0), arm64; Python 3.15.0a8, standard library only; no Docker; both clocks — the time-of-day clock with its NTP step, and the monotonic clock that ignores the step — are modeled deterministically so the whole transcript reproduces byte-for-byte every run. The monotonic clock's absolute value (918273.000) is an arbitrary tick count (uptime-since-boot), deliberately unrelated to wall time — you cannot read civil time off it. That is the point: a monotonic clock has no meaningful absolute value, only a reliable difference, which is the exact +2.000 s interval.

Mental model

Two clocks answer two different questions. Use each for its job.

Both let you write elapsed = end - start. Only one keeps that subtraction honest when the clock is corrected. The whole thing is one script, code/monotonic_vs_wallclock.py.

Setup

cd study/ddia/ch09/code
python3 monotonic_vs_wallclock.py

Do not read the output yet — make each prediction first.


Step 1 · measure a duration with the wall clock, across an NTP step

Predict. We read start from a time-of-day clock, 2 s of work passes, then NTP decides the clock is 5 s fast and steps it back 5 s, then we read end. What does elapsed = end - start come out to?

What does elapsed = end − start come out to?
wall clock (time-of-day, mock so the NTP step reproduces): start = clock.time() -> 1715774400.000 ... 2 s of work happens, clock advances to 1715774402.000 ... NTP steps the clock BACK 5 s -> 1715774397.000 end = clock.time() -> 1715774397.000 elapsed = end - start -> -3.000 s

−3.000 s — a negative duration. The clock ran forward 2 s (to …402) then NTP yanked it back 5 s (to …397), so end is earlier than start. Nothing errored; the subtraction just produced something no stopwatch could.

Step 2 · measure the same interval with the monotonic clock

Predict. The identical modeled interval — 2 s of work, then the same NTP step — but timed by a monotonic clock. The step arrives, but a monotonic clock has no external authority to jump it. What does elapsed come out to?

What does the monotonic elapsed come out to?
monotonic clock (mock, no external authority -- ignores NTP): start = clock.time() -> 918273.000 ... 2 s of work happens, clock advances to 918275.000 ... NTP steps BACK 5 s -> IGNORED, clock stays 918275.000 end = clock.time() -> 918275.000 elapsed = end - start -> +2.000 s

+2.000 s — the true interval. The absolute value 918273.000 is an arbitrary tick count (uptime), deliberately not a wall-clock time — you can't read civil time off it. But the clock only counts up, and the NTP step is invisible to it (step() is a no-op), so end − start recovers exactly the 2 s of work that actually elapsed. This is the same interval Step 1 measured; only the clock differs.

Step 3 · run the identical lease check on each duration

Predict. A lease says "release me once more than 10 s have elapsed": if elapsed > 10: release. Run it on the wall-clock −3.000 s and on the monotonic +2.000 s. Which lease looks expired?

Which lease looks expired?
the same lease check on each measured duration: lease check (wall clock): elapsed -3.000 s > 10 s ? False -> still valid, KEEP holding lease check (monotonic ): elapsed +2.000 s > 10 s ? False -> still valid, KEEP holding verdict: wall-clock elapsed = -3.000 s (NEGATIVE -- impossible for a real duration) monotonic elapsed = +2.000 s (>= 0, the true 2 s interval) a negative duration makes 'elapsed > lease' false forever: the lease never looks expired, so the holder never releases it.

Both read False — but for opposite reasons. The monotonic check is correctly False: exactly 2 s passed, the lease genuinely hasn't hit 10 s, so holding is right. The wall-clock check is silently broken: −3 > 10 is False too, but it would stay False for any lease window — a negative duration can never exceed a positive threshold. The holder waits for an expiry that arithmetic guarantees will never come. (Had the step gone the other way, elapsed would balloon and the lease would expire instantly instead.)


What you should see

Why

From first principles the two clocks are different kinds of quantity. A time-of-day clock is a synchronized value: it is supposed to agree with an external authority (UTC, delivered by NTP), and the price of that agreement is that the authority may correct it in either direction at any time — forward when the clock is slow, back when it is fast, and abruptly (a step) when the error is too large to slew away gently. So two readings taken across a correction are drawn from different calibrations of the same dial; subtracting them is meaningless, and when the correction is backward and larger than the real elapsed time, the difference goes negative. That is exactly what Step 1 shows: +2 s of real work minus a −5 s step leaves −3 s.

A monotonic clock is the opposite: a local counter with no external authority and no obligation to match anyone. Nothing is allowed to reset it, so by construction each reading is >= the previous one and every difference is a valid, non-negative duration. Its absolute value is worthless — you can't tell what time it is from 918273.000 — but duration was the only thing you wanted, and duration is precisely what it protects. The NTP step lands on the wall clock and passes straight through the monotonic one, which is why the same interval reads −3.000 s on one and +2.000 s on the other.

The bug in Step 3, then, is a category error: measuring a duration (how long) with a clock built to report an instant (what time). elapsed = end - start conflates the two — it looks like a duration but inherits the wall clock's power to jump. The fix is not more careful NTP or a sanity check for negatives; it is to read durations from the clock that answers the duration question. Java's System.nanoTime(), POSIX CLOCK_MONOTONIC, Python's time.monotonic() all exist for this one reason.

The boundary — a slew may survive; a step won't, and you can't tell which you'll get NTP usually slews small errors (nudging the clock's rate by a few parts per million) rather than stepping them, and across a slew a short duration is only slightly distorted, not reversed — so a wall-clock duration often looks fine, which is exactly why the bug hides in testing. But whether an error is slewed or stepped depends on its size and the daemon's threshold, and a leap second, a VM resume, or a manual date set will all step the clock regardless. You cannot rely on never being stepped, so the rule is unconditional: measure durations, timeouts, and leases with a monotonic clock.

Go deeper

Sources: DDIA 2e, Ch. 9, §"Monotonic Versus Time-of-Day Clocks", §"Relying on Synchronized Clocks" · Python time module documentation.