For a right-skewed latency distribution, the arithmetic mean lands above the median but far below p99. It overstates the typical request and understates the tail at the same time: 69% of requests are faster than "average," yet p99 is 6× the mean.
"Average response time" sounds like the typical experience. It isn't. For a right-skewed latency distribution the arithmetic mean is dragged up by the long tail, so it lands above the median — yet still far below a high percentile like p99. A single "average" therefore does two wrong things at once: it overstates the typical request and understates the tail. This exercise measures exactly where the mean falls in the sorted request order. You will guess "about half of requests are slower than average" and watch it come back 69%.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — Defining Nonfunctional Requirements, §"Average, Median, and Percentiles":
Usually it is better to use percentiles … the mean is not a very good metric if you want to know your "typical" response time, because it doesn't tell you how many users actually experienced that delay.
The book says prefer percentiles over the mean. Here you measure why — how far the mean drifts from "typical," and how far below the tail it sits.
This exercise is one fact about skewed distributions, made concrete. It helps to be comfortable with these; each line notes where it shows up.
mean > median always.lognormal(mu, sigma), mean = exp(mu + sigma²/2) and P(X < mean) = Φ(sigma/2). These closed forms are the analytic check, and are why the rank depends only on spread.Φ(sigma/2) check. You only need "Φ turns a z-score into a percentile."exp(mu + sigma²/2) mean and the CDF in terms of Φ. If only two are new, make them #1 and #2 — they carry the whole result.
random.Random(42), so your run reproduces these exact figures. A different arch/version shifts the digits slightly; the shape holds.
We model request latency as the same right-skewed lognormal used in the tail-latency exercise: median 10 ms, p99 ≈ 100 ms (mu = ln 10 ≈ 2.303, sigma ≈ 0.990). We draw a pool of 1,000,000 requests, then ask two things a single "average" cannot answer:
p99 / mean — how badly a reported "average" undersells the request 1 user in 100 actually gets.The whole exercise is one script, code/mean_hides_tail.py (~80 lines of stdlib). Read it once, then run it and check each prediction below.
cd study/ddia/code
# no dependencies; just Python 3
python3 mean_hides_tail.py # ~2 s: draws 1,000,000 samplesDo not read the output yet — make each prediction first.
Predict. The distribution has median 10 ms. Before looking: roughly what is its mean? (Most people say "also around 10 ms — maybe 11 or 12.")
python3 mean_hides_tail.py # read the first blockThe mean is 16.3 ms — 1.63× the median. The tail has already pulled "average" 63% above the actual midpoint, and we haven't even asked the interesting question yet.
Predict. Sort all million requests slowest-to-fastest. What fraction of them are faster than the mean (16.3 ms)? Write a number down. (The instinct is 50% — "half are above average, half below.")
69%. More than two in three requests are faster than "average." The mean is not the typical experience — the typical (median) request is 10 ms, and the average request is slower than roughly 7 out of 10 of them. The simulation (69.0%) lands exactly on the analytic Φ(sigma/2) = 69.0%.
Look at the ratios again (no new run needed).
Predict. The mean is 16.3 ms. How many times larger is p99?
p99 / mean = 6.11× (and p99.9 is 13×). So the same single number that overstated the typical request (69% are faster) also understated the tail sixfold. One user in 100 waits >6× the "average"; one in 1,000 waits >13×. The mean is blind in both directions at once.
Φ(sigma/2)).The rank of the mean (69%) depends only on the spread sigma, not on the median — halve sigma and the mean marches back toward p50; widen it and the mean ranks even higher.
The mean and the median answer different questions, and only one of them is robust to a tail. The median is rank-based: it is the latency of the middle-ranked request and does not care whether the slowest request took 200 ms or 2,000 ms. The mean is sum-based: every tail value contributes its full magnitude to the total, so a handful of 200 ms requests drag the average upward even though they are rare. In any right-skewed distribution this makes mean > median — necessarily, not coincidentally.
The size of the gap is exact for a lognormal. The mean is exp(mu + sigma²/2) = median × exp(sigma²/2) = 10 × exp(0.49) = 16.3 ms. And the rank of the mean is P(X < mean) = Φ(sigma/2) = Φ(0.495) = 69% — notice this depends only on sigma (the spread), not on the median at all. Fatter tail, higher rank: the mean drifts further from "typical" the more skewed the traffic is. Meanwhile the tail keeps climbing well past it — p99 is 6× the mean — so the one number lands in a no-man's-land: too high to be typical, too low to be the tail.
mean = median and the mean would sit exactly at p50, faster than 50% of requests — reporting the average would be fine. It is the shape of latency, not the existence of an average, that makes the mean misleading. That is why DDIA's advice is specific: for response times — which are essentially always right-skewed — report percentiles (p50, p95, p99), because they answer "how many users actually experienced this?" and the mean does not.
SIGMA. Predict where the mean ranks with a thinner tail, then edit SIGMA and re-run. Φ(sigma/2) with sigma ≈ 0.495 predicts ~59% — the mean creeps back toward the median as skew shrinks.Sources: DDIA 2e, Ch. 2, §"Average, Median, and Percentiles" · the lognormal mean exp(mu + sigma²/2) and rank Φ(sigma/2) are standard results (see "Log-normal distribution", Wikipedia).