# DDIA Ch. 2: the mean hides the tail ## Concept "Average response time" sounds like the typical experience. It isn't. For a right-skewed latency distribution the arithmetic mean is dragged *up* by the long tail, so it lands **above the median** — yet still far **below** a high percentile like p99. A single "average" therefore does two wrong things at once: it overstates the typical request *and* understates the tail. This exercise measures exactly where the mean falls in the sorted request order. You will guess "about half of requests are slower than average" and watch it come back **69%**. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — *Defining Nonfunctional Requirements*, §"Average, Median, and Percentiles": > Usually it is better to use percentiles … the mean is not a very good metric > if you want to know your "typical" response time, because it doesn't tell you > how many users actually experienced that delay. The book says prefer percentiles over the mean. Here you measure *why* — how far the mean drifts from "typical," and how far below the tail it sits. ## Prerequisites This exercise is one fact about skewed distributions, made concrete. It helps to be comfortable with these; each line notes where it shows up. 1. **Mean vs. median** — the mean is a *sum*-based center (every value contributes its full magnitude); the median is a *rank*-based center (only the middle-ranked value matters). This difference is the whole exercise. 2. **Skew and the long tail** — a right-skewed distribution has a few very large values. They pull the mean up but leave the median where it is, so `mean > median` always. Explains why "average" exceeds the midpoint. 3. **Percentiles & percentile rank** — a pX is the value X% of samples fall below; the *rank* of a value is the % of samples below it. We measure the rank of the mean. 4. **The lognormal's mean** — for `lognormal(mu, sigma)`, `mean = exp(mu + sigma²/2)` and `P(X < mean) = Φ(sigma/2)`. These two closed forms are the analytic check at the end, and #4 is why the rank depends only on spread. 5. **The normal CDF Φ** — the standard-normal cumulative function used in that `Φ(sigma/2)` check. You only need "Φ turns a z-score into a percentile." ### Where to learn them - **Mean, median, skew (#1–#2):** Khan Academy's *Statistics* unit (measures of center, skew) is the quickest path; for depth, Blitzstein & Hwang's *Introduction to Probability* (Harvard Stat 110), free textbook + videos (search "Stat 110" / stat110.net). - **Percentiles in latency (#3):** Gil Tene, "How NOT to Measure Latency" (talk on YouTube/InfoQ) — why the mean misleads and percentiles don't. - **The lognormal and Φ (#4–#5):** the Wikipedia "Log-normal distribution" article gives the `exp(mu + sigma²/2)` mean and the CDF in terms of Φ. If only two are new, make them #1 and #2 — they carry the whole result. ## Environment Every number below was captured on this exact environment. Re-run elsewhere and the digits shift slightly; the *shape* holds: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only (no numpy) - `random.Random(42)` — fixed seed, so your run reproduces these figures ## Mental model We model request latency as the same right-skewed **lognormal** used in the tail-latency exercise: median 10 ms, p99 ≈ 100 ms (`mu = ln 10 ≈ 2.303`, `sigma ≈ 0.990`). We draw a pool of 1,000,000 requests, then ask two things a single "average" cannot answer: - **Where does the mean rank?** Sort every request. What fraction are *faster* than the mean? For a symmetric distribution the answer would be 50% (the mean sits at the median). Skew moves it. - **How far is the tail above the mean?** The ratio `p99 / mean` — how badly a reported "average" undersells the request 1 user in 100 actually gets. The whole exercise is one script, `code/mean_hides_tail.py` (~80 lines of stdlib). Read it once, then run it and check each prediction below. ## Setup ``` cd study/ddia/ch02/code # no dependencies; just Python 3 python3 mean_hides_tail.py # ~2 s: draws 1,000,000 samples ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — mean vs. median **Predict.** The distribution has median 10 ms. Before looking: roughly what is its *mean*? (Most people say "also around 10 ms — maybe 11 or 12.") **Do.** Run the script; read the first block. **Observe.** ``` request latency (n=1,000,000, lognormal mu=2.303 sigma=0.990): mean = 16.3 ms p50 = 10.0 ms (median -- the true midpoint) p90 = 35.6 ms p99 = 99.6 ms p999 = 212.4 ms ``` The mean is **16.3 ms — 1.63× the median.** The tail has already pulled "average" 63% above the actual midpoint, and we haven't even asked the interesting question yet. ### Step 2 — where the mean ranks (the surprise) **Predict.** Sort all million requests slowest-to-fastest. What fraction of them are *faster* than the mean (16.3 ms)? Write a number down. (The instinct is 50% — "half are above average, half below.") **Do.** Read the second block. **Observe.** ``` what the single 'average' number hides: mean / median = 1.63x (mean is pulled above the midpoint) % of requests faster than the mean = 69.0% p99 / mean = 6.11x (the tail the mean doesn't show) p999 / mean = 13.03x analytic check: mean = exp(mu + sigma^2/2) = 16.3 ms analytic check: P(X < mean) = Phi(sigma/2) = 69.0% ``` **69%.** More than two in three requests are *faster* than "average." The mean is not the typical experience — the typical (median) request is 10 ms, and the average request is slower than roughly 7 out of 10 of them. The simulation (69.0%) lands exactly on the analytic `Φ(sigma/2) = 69.0%`. ### Step 3 — the tail the mean doesn't show Look at the ratios again (no new run needed). **Predict.** The mean is 16.3 ms. How many times larger is p99? **Observe.** **`p99 / mean = 6.11×`** (and p99.9 is **13×**). So the same single number that *overstated* the typical request (69% are faster) also *understated* the tail sixfold. One user in 100 waits >6× the "average"; one in 1,000 waits >13×. The mean is blind in both directions at once. ## What you should see - Mean ≈ **16.3 ms**, median = **10.0 ms** → mean is **1.63×** the median. - **69%** of requests are faster than the mean (matches `Φ(sigma/2)`). - **p99 / mean ≈ 6.1×**, **p999 / mean ≈ 13×**. The rank of the mean (69%) depends only on the spread `sigma`, not on the median — halve `sigma` and the mean marches back toward p50; widen it and the mean ranks even higher. ## Why The mean and the median answer different questions, and only one of them is robust to a tail. The **median** is rank-based: it is the latency of the middle-ranked request and does not care whether the slowest request took 200 ms or 2,000 ms. The **mean** is sum-based: every tail value contributes its full magnitude to the total, so a handful of 200 ms requests drag the average upward even though they are rare. In any right-skewed distribution this makes `mean > median` — necessarily, not coincidentally. The size of the gap is exact for a lognormal. The mean is `exp(mu + sigma²/2) = median × exp(sigma²/2) = 10 × exp(0.49) = 16.3 ms`. And the *rank* of the mean is `P(X < mean) = Φ(sigma/2) = Φ(0.495) = 69%` — notice this depends only on `sigma` (the spread), not on the median at all. Fatter tail, higher rank: the mean drifts further from "typical" the more skewed the traffic is. Meanwhile the tail keeps climbing well past it — p99 is 6× the mean — so the one number lands in a no-man's-land: too high to be typical, too low to be the tail. **The flip side — no skew, no problem.** This is purely a consequence of right-skew. If latency were symmetric (a normal distribution), `mean = median` and the mean would sit exactly at p50, faster than 50% of requests — reporting the average would be fine. It is the *shape* of latency, not the existence of an average, that makes the mean misleading. That is why DDIA's advice is specific: for response times — which are essentially always right-skewed — report percentiles (p50, p95, p99), because they answer "how many users actually experienced this?" and the mean does not. ### Go deeper 1. **Halve `SIGMA`.** Predict where the mean ranks with a thinner tail, then edit `SIGMA` and re-run. `Φ(sigma/2)` with `sigma ≈ 0.495` predicts ~59% — the mean creeps back toward the median as skew shrinks. 2. **Report what an SLA would.** Add p95 to the output. A team that promises "average < 20 ms" is trivially meeting it (16.3 ms) while 1% of users wait >99 ms. Which number would you put in the SLA, and why? 3. **Mean of the max.** Combine with the tail-latency exercise: for an N=100 fan-out request, is the *mean* end-user latency a useful number, or does the same blindness get worse? Predict, then measure.