study › DDIA › Chapter 2 · Nonfunctional Requirements

Tail-Latency Amplification

One backend call is slow just 1% of the time. Fan a request out to 100 of them in parallel and wait for the slowest, and 63% of end-user requests cross a line only 1 in 100 individual calls ever cross — and the median request ends up slower than the single call's 99th percentile.

Concept

A backend call that is fast 99% of the time feels fast. But an end-user request rarely hits one backend — it fans out to many, in parallel, and cannot return until the slowest one does. This exercise measures what that does: a latency threshold only 1 in 100 individual calls ever crosses gets crossed by nearly two-thirds of end-user requests, purely because each request rolls the dice 100 times. You will predict "still ~99% fast" and watch it come back 37%.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — Defining Nonfunctional Requirements, §"Use of Response Time Metrics":

High percentiles are especially important in backend services that are called multiple times as part of serving a single end-user request. Even if you make the calls in parallel, the request still needs to wait for the slowest of the parallel calls to complete. It takes just one slow call to make the entire end-user request slow … an effect known as tail latency amplification.

The book states the effect. Here you produce it and measure its size.

Prerequisites

The surprise here is entirely a probability result — the code is just a witness to it. You'll get the most from the exercise if these ideas are comfortable; each notes exactly where it shows up below.

Independent events & the multiplication rule — for independent events, P(A and B) = P(A)·P(B). The whole ballgame: a request is fast only if every call is fast, and multiplying 0.99 by itself 100 times is what produces 0.366.
The complement rule — P(at least one slow) = 1 − P(none slow). Turns an awkward "at least one" into an easy "1 minus all-fast."
Percentiles / quantiles — p50, p90, p99, p99.9. A pX is the value X% of samples fall below. The "slow line" is a percentile.
The small-p exponential approximation — (1−p)^N ≈ e^(−pN) for small p. Here pN = 0.01 × 100 = 1, so the fast fraction ≈ e⁻¹ ≈ 0.37. That "1" is the expected number of slow calls per request.
Distribution shape / the lognormal — right-skewed, median ≠ mean, long right tail. Why a "usually 10 ms" service still throws 200 ms calls.
The maximum of N draws (order statistics) — a fan-out request's latency is the max of its parallel calls; sampling the max repeatedly drags you into the tail.
Monte Carlo estimation — approximating a probability by simulating many trials; why the simulated 63.4% and analytic 63.4% line up (and differ by a hair).

If only two of these are new, make them #1 and #2 — they carry 90% of the result. #4 is the shortcut that gets you "37%" in your head.

Where to learn the prerequisites Independence, complement rule, percentiles (the core): Khan Academy's Probability unit is the fastest path to just the rules used here; for a real foundation, Blitzstein & Hwang's Introduction to Probability (Harvard's Stat 110) — free textbook and lecture videos (search "Stat 110" / stat110.net). Why e shows up: 3Blue1Brown's binomial and e videos, for #4 and #6. Percentiles & latency: Gil Tene, "How NOT to Measure Latency" (talk on YouTube/InfoQ). The lognormal: the Wikipedia "Log-normal distribution" article is enough. The fan-out result in the wild: Dean & Barroso, "The Tail at Scale" (CACM, 2013).

Environment these numbers came from Every figure below was captured on macOS 26.5.2 (Darwin 25.5.0), arm64, with Python 3.15.0a8 (standard library only). The script seeds random.Random(42), so your run reproduces these exact figures. A different arch/version shifts the digits slightly; the shape holds.

Mental model

We model one backend call's latency as a right-skewed lognormal distribution, tuned so the median is 10 ms and the 99th percentile is 100 ms — a realistic "usually fast, occasionally a long tail" service. Call anything past that p99 (≈100 ms) a slow call: by construction, exactly 1% of individual calls are slow.

An end-user request fans out to N of these calls in parallel and returns only when the last one finishes, so its latency is max(N draws). Two quantities move as N grows:

Fraction slow — the chance a request contains at least one slow call. Each call is slow independently with probability p = 0.01, so by the multiplication rule a request is not slow only if all N calls are fast: P(fast) = (1-p)^N; and by the complement rule P(slow) = 1 - (1-p)^N. A shortcut worth carrying: for small p, (1-p)^N ≈ e^(-pN), where pN is the expected number of slow calls per request. At N=100, pN = 0.01 × 100 = 1 exactly, so the fast fraction ≈ e⁻¹ ≈ 37%, leaving ~63% slow — that is where the "37%" in the Concept comes from. That number falls off a cliff you would not guess by intuition.
The percentiles of the max — the median and p99 of the end-user request, which drift far above the single-call figures because you are repeatedly sampling the tail.

The whole exercise is one script, code/tail_latency.py (~90 lines of stdlib). Read it once, then run it and check each prediction against its output.

Setup

cd study/ddia/ch02/code
# no dependencies; just Python 3
python3 tail_latency.py     # ~10 s: draws 1,000,000 samples + simulates

Do not read the output yet — make each prediction first.

Step 1 · the single call (calibration)

Predict. One backend call, lognormal, median 10 ms. Before looking: roughly what is its p90? Its p99? These set the "slow" line for everything after.

python3 tail_latency.py   # read the first block

Predict the p90 and p99 of a single call

single backend call (n=1,000,000, lognormal mu=2.303 sigma=0.990): p50 = 10.0 ms p90 = 35.6 ms p99 = 99.6 ms <- 'slow' threshold: 1 in 100 calls cross it p999= 212.4 ms

The p90 is already 3.5× the median, and p99 is ~10× — the tail is fat, but 99% of calls still come back under ~100 ms. So far, nothing alarming. Fix 99.6 ms in your head as the slow line: only 1 call in 100 is worse.

Step 2 · the amplification (the surprise)

Now the core claim. Each backend call is slow (>99.6 ms) just 1% of the time. An end-user request fans out to N=100 such calls in parallel and waits for the slowest.

Predict. What fraction of end-user requests come back slower than 99.6 ms — the line only 1% of individual calls cross? Write a number down. (Most people say something near 1%, or hedge up to "maybe 5–10%.")

Guess the fraction of fan-out requests slower than the single-call p99

end-user request = max of N parallel calls (n=200,000 requests each): N p50 p99 % slower than single-call p99 1 10.0ms 101.8ms 1.1% 10 44.3ms 210.7ms 9.5% 100 114.1ms 399.0ms 63.4% analytic check: P(>=1 slow call | N=100) = 1 - 0.99^100 = 63.4%

63.4%. A threshold that 1 in 100 individual calls crosses is crossed by nearly two in three end-user requests, once each request depends on 100 of them. The simulation (63.4%) lands exactly on the analytic 1 - 0.99^100 = 63.4%.

Step 3 · the median moved past the old p99

Look at the p50 column again, across the same table (no new run needed).

Predict. Fan-out doesn't change any individual call — so does the median end-user latency stay around 10 ms?

Does the median end-user latency stay near 10 ms at N=100?

At N=100 the median end-user request is 114.1 ms — already slower than the single call's 99th percentile (99.6 ms). The typical request under fan-out is worse than the worst-1% of the backend it's built from. p99 climbs from ~100 ms to ~399 ms over the same span.

What you should see

Single-call p99 ≈ 99.6 ms (the slow line), p50 = 10 ms.
Fraction of end-user requests past that line: 1.1% → 9.5% → 63.4% for N = 1, 10, 100 — tracking 1-(1-0.01)^N.
Median end-user latency at N=100 (114 ms) exceeds the single-call p99.

Why the simulated 9.5% isn't exactly the analytic 9.6% The "slow" threshold is the empirical p99 of a finite 1,000,000-sample pool, not exactly the 1.000% point, so the simulated fraction sits a hair below the analytic at N=10. Monte-Carlo honesty, not a bug — the transcript is the raw run.

Why

The math is unforgiving because the failures compound multiplicatively, not additively. Intuition averages ("1% slow, so ~1% slow") because we model the request as one thing. But a fan-out request is an AND of independent bets: it is fast only if call 1 AND call 2 AND … AND call 100 are all fast. Independent probabilities under AND multiply, and 0.99^100 ≈ 0.366 — so ~63% fail the all-fast test. This is why DDIA says high percentiles matter most in exactly the services called many times per request: the more a request leans on, the more its latency is governed by the tail of its dependencies, not their median.

The flip side — one backend, no amplification This effect is entirely a property of fan-out. If an end-user request touches exactly one backend, its slow rate is that backend's slow rate: 1%, full stop. Nothing amplifies. The 63% appears only because the request's fate is an AND over many independent calls — remove the "many" and the AND collapses to a single term (0.99^1). So the lesson isn't "backends are slow," it's "depending on many backends at once turns a rare per-call event into a common per-request one." A service behind a single backend can read its p99 straight off; a request that fans out to 100 cannot — which is why this bites search shards, microservice graphs, and scatter-gather queries, and not a plain single-service call.

It also reframes what "p99" is for. A 1-in-100 backend event isn't rare from the perspective of a request that makes 100 backend calls — it's the common case. Amazon famously specs internal services at the p99.9, even though it touches 1 in 1,000 requests, precisely because fan-out drags that tail into everyday user experience.

Go deeper

Push N to 200. Predict the fraction first, then edit FANOUTS to include 200 (or type python3 -c "print(1-0.99**200)"). It's 86.6% — the all-fast case is now the exception.
The fix: hedged requests. DDIA and the Dean & Barroso "tails at scale" work describe sending the same call to two backends and taking whichever returns first — latency becomes min of two draws instead of one. Add a variant that, per backend call, takes min(two draws) and re-measure the N=100 fraction. Predict how far 63% drops before you run it.
Change the tail, not the median. Halve SIGMA (thinner tail, same 10 ms median) and re-run. The median barely moves; the amplified fraction collapses — evidence that fan-out latency is a tail problem, and that optimizing the median is the wrong lever.

Sources: DDIA 2e, Ch. 2, §"Use of Response Time Metrics" and §"Average, Median, and Percentiles" · Dean & Barroso, "The Tail at Scale", CACM 2013 — the source of the fan-out and hedged-request analysis.