# DDIA Ch. 2: tail-latency amplification ## Concept A backend call that is fast 99% of the time feels fast. But an end-user request rarely hits *one* backend — it fans out to many, in parallel, and cannot return until the **slowest** one does. This exercise measures what that does: a latency threshold only 1 in 100 individual calls ever crosses gets crossed by nearly **two-thirds** of end-user requests, purely because each request rolls the dice 100 times. You will predict "still ~99% fast" and watch it come back 37%. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — *Defining Nonfunctional Requirements*, §"Use of Response Time Metrics": > High percentiles are especially important in backend services that are > called multiple times as part of serving a single end-user request. Even > if you make the calls in parallel, the request still needs to wait for the > slowest of the parallel calls to complete. It takes just one slow call to > make the entire end-user request slow … an effect known as **tail latency > amplification**. The book states the effect. Here you produce it and measure its size. ## Prerequisites The surprise in this exercise is entirely a probability result — the code is just a witness to it. You'll get the most out of it if these seven ideas are comfortable. Each line notes exactly where it shows up below. 1. **Independent events & the multiplication rule** — for independent events, `P(A and B) = P(A)·P(B)`. This is the whole ballgame: a request is fast only if *every* call is fast, and multiplying 0.99 by itself 100 times is what produces 0.366. 2. **The complement rule** — `P(at least one slow) = 1 − P(none slow)`. Turns the awkward "at least one" into an easy "1 minus all-fast." 3. **Percentiles / quantiles** — p50 (median), p90, p99, p99.9. A pX is the value that X% of samples fall below. The entire "slow line" is a percentile. 4. **The small-p exponential approximation** — `(1−p)^N ≈ e^(−pN)` when p is small. Here `p·N = 0.01 × 100 = 1`, so the fast fraction ≈ `e⁻¹ ≈ 0.37`. That "1" is the *expected number of slow calls per request* — the number to hold onto. 5. **Distribution shape / the lognormal** — right-skewed, median ≠ mean, long right tail. Explains why a "usually 10 ms" service still throws 200 ms calls. 6. **The maximum of N draws (order statistics)** — a fan-out request's latency is the *max* of its parallel calls; sampling the max repeatedly is what drags you into the tail. 7. **Monte Carlo estimation** — approximating a probability by simulating many trials. It's why the simulated 63.4% and the analytic 63.4% should line up (and why they differ by a hair — see "What you should see"). If only two of these are new to you, make them #1 and #2 — they're 90% of the result. #4 is the shortcut that lets you get "37%" in your head. ### Where to learn them - **Independence, complement rule, percentiles (the core):** Khan Academy's *Probability* unit is the fastest path to just the rules used here. For a real foundation, **Blitzstein & Hwang, *Introduction to Probability*** (aka Harvard's Stat 110) — the textbook PDF and the full lecture videos are free (search "Stat 110" / stat110.net). - **Why `e` shows up / the exponential approximation:** **3Blue1Brown** on YouTube — the binomial-distribution and `e` videos give the visual intuition for #4 and #6. - **Percentiles & latency specifically:** **Gil Tene, "How NOT to Measure Latency"** (talk on YouTube/InfoQ) — why averages lie and what p99/p99.9 actually mean. Directly on-topic for this exercise. - **The lognormal:** the Wikipedia "Log-normal distribution" article is enough — you only need "right-skewed, parameterized by the mu/sigma of an underlying normal." - **The fan-out result in the wild:** **Dean & Barroso, "The Tail at Scale"** (CACM, 2013) — the paper this exercise operationalizes (also cited in "Go deeper"). ## Environment Every number below was captured on this exact environment. Re-run on a different one and the transcripts will differ slightly (the *shape* holds): - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only (no numpy) - `random.Random(42)` — fixed seed, so your run reproduces these figures ## Mental model We model one backend call's latency as a right-skewed **lognormal** distribution, tuned so the median is 10 ms and the 99th percentile is 100 ms — a realistic "usually fast, occasionally a long tail" service. Call anything past that p99 (≈100 ms) a **slow** call: by construction, exactly 1% of individual calls are slow. An end-user request fans out to *N* of these calls in parallel and returns only when the last one finishes, so its latency is `max(N draws)`. Two quantities move as *N* grows: - **Fraction slow** — the chance a request contains *at least one* slow call. Each call is slow independently with probability *p* = 0.01, so by the **multiplication rule** a request is *not* slow only if *all N* calls are fast: `P(fast) = (1-p)^N`; and by the **complement rule** `P(slow) = 1 - (1-p)^N`. A shortcut worth carrying: for small *p*, `(1-p)^N ≈ e^(-pN)`, where `pN` is the *expected number of slow calls per request*. At N=100, `pN = 0.01 × 100 = 1` exactly — one expected slow call per request — so the fast fraction is ≈ `e⁻¹ ≈ 37%`, leaving ~63% slow. (That is where the "37%" in the Concept comes from.) That number falls off a cliff you would not guess by intuition. - **The percentiles of the max** — the median and p99 of the end-user request, which drift far above the single-call figures because you are repeatedly sampling the tail. The whole exercise is one script: `code/tail_latency.py`. Read it once (it's ~90 lines of stdlib), then run it and check each prediction below against its output. ## Setup ``` git clone && cd study/ddia/ch02/code # no dependencies; just Python 3 python3 tail_latency.py # ~10 s: it draws 1,000,000 samples + simulates ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — the single call (calibration) **Predict.** One backend call, lognormal, median 10 ms. Before looking: roughly what is its p90? Its p99? (These set the "slow" line for everything after.) **Do.** Run the script; read the first block. **Observe.** ``` single backend call (n=1,000,000, lognormal mu=2.303 sigma=0.990): p50 = 10.0 ms p90 = 35.6 ms p99 = 99.6 ms <- 'slow' threshold: 1 in 100 calls cross it p999= 212.4 ms ``` The p90 is already 3.5× the median, and p99 is ~10× — the tail is fat, but 99% of calls still come back under ~100 ms. So far, nothing alarming. Fix **99.6 ms** in your head as the slow line: only 1 call in 100 is worse. ### Step 2 — the amplification (the surprise) Now the core claim. Each backend call is slow (>99.6 ms) just 1% of the time. An end-user request fans out to **N=100** such calls in parallel and waits for the slowest. **Predict.** What fraction of *end-user requests* come back slower than 99.6 ms — the line only 1% of individual calls cross? Write a number down before revealing. (Most people say something near 1%, or hedge up to "maybe 5–10%.") **Do.** Read the fan-out table. **Observe.** ``` end-user request = max of N parallel calls (n=200,000 requests each): N p50 p99 % slower than single-call p99 1 10.0ms 101.8ms 1.1% 10 44.3ms 210.7ms 9.5% 100 114.1ms 399.0ms 63.4% analytic check: P(>=1 slow call | N=100) = 1 - 0.99^100 = 63.4% ``` **63.4%.** A threshold that 1 in 100 individual calls crosses is crossed by nearly two in three end-user requests, once each request depends on 100 of them. The simulation (63.4%) lands exactly on the analytic `1 - 0.99^100 = 63.4%`. ### Step 3 — the median moved past the old p99 Look at the `p50` column again, across the same table (no new run needed). **Predict.** Fan-out doesn't change any individual call — so does the *median* end-user latency stay around 10 ms? **Observe.** At **N=100 the median end-user request is 114.1 ms** — already *slower than the single call's 99th percentile* (99.6 ms). The typical request under fan-out is worse than the worst-1% of the backend it's built from. p99 climbs from ~100 ms to ~399 ms over the same span. ## What you should see - Single-call p99 ≈ **99.6 ms** (the slow line), p50 = 10 ms. - Fraction of end-user requests past that line: **1.1% → 9.5% → 63.4%** for N = 1, 10, 100 — tracking `1-(1-0.01)^N`. - Median end-user latency at N=100 (**114 ms**) exceeds the single-call p99. The simulated fraction (9.5%) sits a hair below the analytic (9.6%) at N=10 because the "slow" threshold is the *empirical* p99 of a finite pool, not exactly the 1.000% point — Monte-Carlo honesty, not a bug. ## Why The math is unforgiving because the failures **compound multiplicatively**, not additively. Intuition averages ("1% slow, so ~1% slow") because we model the request as *one* thing. But a fan-out request is an AND of independent bets: it is fast only if call 1 AND call 2 AND … AND call 100 are all fast. Independent probabilities under AND multiply, and `0.99^100 ≈ 0.366` — so ~63% fail the all-fast test. This is why DDIA says high percentiles matter *most* in exactly the services called many times per request: the more a request leans on, the more its latency is governed by the tail of its dependencies, not their median. **The flip side — one backend, no amplification.** This effect is *entirely* a property of fan-out. If an end-user request touches exactly one backend, its slow rate is that backend's slow rate: 1%, full stop. Nothing amplifies. The 63% appears only because the request's fate is an AND over *many* independent calls — remove the "many" and the AND collapses to a single term (`0.99^1`). So the lesson isn't "backends are slow," it's "depending on many backends at once turns a rare *per-call* event into a common *per-request* one." A service sitting behind a single backend can read its p99 straight off; a request that fans out to 100 cannot — which is exactly why this matters for search shards, microservice graphs, and scatter-gather queries, and not for a plain single-service call. It also reframes what "p99" is *for*. A 1-in-100 backend event isn't rare from the perspective of a request that makes 100 backend calls — it's the common case. Amazon famously specs internal services at the **p99.9**, even though it touches 1 in 1,000 requests, precisely because fan-out drags that tail into everyday user experience. ### Go deeper 1. **Push N to 200.** Predict the fraction first, then edit `FANOUTS` to include 200 (or type `python3 -c "print(1-0.99**200)"`). It's **86.6%** — the all-fast case is now the exception. 2. **The fix: hedged requests.** DDIA and the Dean & Barroso "tails at scale" work describe sending the same call to two backends and taking whichever returns first — latency becomes `min` of two `max`-style draws instead of `max`. Add a variant to the script that, per backend call, takes `min(two draws)` and re-measure the N=100 fraction. Predict how far 63% drops before you run it. 3. **Change the tail, not the median.** Halve `SIGMA` (thinner tail, same 10 ms median) and re-run. The median barely moves; the amplified fraction collapses — evidence that fan-out latency is a tail problem, and that optimizing the median is the wrong lever.