# DDIA Ch. 2: averaging percentiles is a lie ## Concept Every machine behind a load balancer reports its own p99. The obvious way to get a single "fleet p99" is to average those per-machine numbers. That average is **mathematically meaningless** — it is not the p99 of anything. This exercise builds a 10-server fleet with one degraded machine, averages the per-server p99s, and compares that to the *true* p99 of all requests pooled together. You will predict "close enough" and watch the averaged number land **21% below** the real tail — with **78%** of that tail coming from a single server that handles only 10% of traffic. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — *Defining Nonfunctional Requirements*, §"Computing Percentiles": > When you have several HTTP calls … you need to be careful about how you > aggregate response times … averaging percentiles is meaningless. The correct > way of aggregating response time data is to add the histograms. The book states the rule. Here you produce the error the rule warns against and measure how big it is. ## Prerequisites The surprise is a fact about what a percentile *is* and what it throws away. These help; each line notes where it shows up. 1. **Percentiles / quantiles** — a pX is the value X% of samples fall below. The entire exercise compares two ways of getting one. 2. **A percentile is a lossy summary** — it reports a single point of a distribution and discards the rest of the shape. This is *why* you can't reassemble the whole from the parts. 3. **Means don't commute with percentiles** — `p99(A ∪ B) ≠ mean(p99(A), p99(B))` in general. The core claim; the whole run is a demonstration of the inequality. 4. **Histograms as the aggregable form** — counts-per-latency-bucket *do* add across machines (bucket-by-bucket), and the pooled percentile is read off the summed histogram. This is the correct method the book prescribes. 5. **Load balancing / pooling** — requests spread across machines; the user-visible tail is a property of the *pool*, not of any one machine. ### Where to learn them - **Percentiles & why the naive aggregation fails (#1–#3):** Gil Tene, "How NOT to Measure Latency" (talk on YouTube/InfoQ) — this exact pitfall, in depth. For fundamentals, Khan Academy's *Statistics* percentiles unit. - **Histograms as the aggregable representation (#4):** read up on **HdrHistogram** (Gil Tene) or **t-digest** (Ted Dunning) — the data structures production systems use precisely so percentiles *can* be merged across machines. Their docs explain why you store histograms, not percentiles. If only one is new, make it #3 — everything else is scaffolding for it. ## Environment Every number below was captured on this exact environment. Re-run elsewhere and the digits shift slightly; the *shape* holds: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only (no numpy) - `random.Random(42)` — fixed seed, so your run reproduces these figures ## Mental model Ten servers sit behind a load balancer, each handling the same number of requests (100,000; 1,000,000 total). Nine are healthy — the same lognormal as the other exercises, median 10 ms, p99 ≈ 100 ms. One (server 0) is **degraded**: 4× slower median (40 ms), same spread, so its whole latency curve is shifted up. Each server computes its own p99. We compare two "fleet p99" numbers: - **Naive** — the mean of the ten per-server p99s. What a dashboard that averages machine metrics would show. - **True** — the p99 of all 1,000,000 requests pooled into one list. This is what "add the histograms" computes, and it is the number a user's tail experience actually follows. The whole exercise is one script, `code/averaging_percentiles.py` (~90 lines of stdlib). Read it once, then run it and check each prediction below. ## Setup ``` cd study/ddia/code # no dependencies; just Python 3 python3 averaging_percentiles.py # ~3 s: simulates 1,000,000 requests ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — the per-server picture **Predict.** Nine healthy servers (p99 ≈ 100 ms) and one degraded server (4× slower). Before looking: roughly what is the degraded server's own p99? **Do.** Run the script; read the per-server block. **Observe.** ``` fleet of 10 servers, 100,000 requests each (1,000,000 total); server 0 is degraded (4x slower): per-server p99 (ms): server 0: 393.3 <- degraded server 1: 99.3 server 2: 97.8 server 3: 98.9 server 4: 99.4 server 5: 100.0 server 6: 101.0 server 7: 100.0 server 8: 100.2 server 9: 101.5 ``` Nine servers cluster at ~100 ms; the sick one is at **393 ms**. Nothing surprising yet — this is exactly the data a per-machine dashboard shows. ### Step 2 — averaging the p99s (the surprise) Now collapse those ten numbers into one "fleet p99" the obvious way: average them. Then compare to the true p99 of every request pooled together. **Predict.** How close is `mean(per-server p99s)` to the true pooled p99? Write down a guess for both numbers. (Most people expect them within a few percent — "it's an average of p99s, it should be about the p99.") **Do.** Read the rest of the output. **Observe.** ``` NAIVE fleet p99 = mean(per-server p99s) = 129.1 ms TRUE fleet p99 = p99(all requests pooled) = 162.9 ms the averaged number is off by -20.7% (understates the real tail) slowest 1% of all requests: 78% of them came from the single degraded server -- which is only 10% of the traffic. ``` The averaged number is **129 ms**; the real p99 is **163 ms**. The average *understates the tail your users actually feel by 21%* — and it is not even a predictable direction (see Go deeper). The two numbers are answers to different questions. ### Step 3 — where the tail actually comes from Look at the last line (no new run needed). **Predict.** The degraded server handles 1 in 10 requests. Of the fleet's slowest 1% overall, what share came from it? **Observe.** **78%.** The one sick server, though only a tenth of traffic, produced more than three-quarters of the slowest 1% of all requests. Averaging the ten p99s divides its spike by ten and throws that away — which is precisely why the naive number misses. ## What you should see - Degraded server p99 ≈ **393 ms**; healthy servers ≈ **100 ms**. - Naive fleet p99 (**129 ms**) vs. true pooled p99 (**163 ms**) → off by **~21%**, understating the tail. - **78%** of the slowest 1% of requests come from the single degraded server. ## Why A percentile is a *lossy* summary: it reports one point of a distribution and discards the rest of its shape. Server 0's "p99 = 393 ms" tells you one number and hides that server 0 also emitted tens of thousands of requests in the 150–400 ms range. The fleet's true p99 is decided by the **10,000 slowest requests out of 1,000,000** — and 78% of those are server 0's. When you average the ten per-server p99s, you compress server 0's entire heavy upper tail into one value and then divide it by ten. You have thrown away exactly the information the pooled p99 depends on, so the average cannot reconstruct it. In symbols: `p99(A ∪ B) ≠ mean(p99(A), p99(B))`, because a percentile of a union is not a function of the percentiles of the parts. The direction of the error isn't even fixed. Here the sick server has *average* volume, so its many slow requests flood the pooled tail and the naive average *understates* the truth. Give it *low* volume instead — a small, very slow node — and its huge p99 still enters the average at weight 1/10 while barely touching the pool, so the naive number would *overstate* the real p99. You cannot predict even the sign of the error from the per-server p99s alone. That is what "meaningless" means: the information required simply isn't in the inputs. The correct aggregation is the one the book names: **add the histograms.** Keep each machine's full latency distribution as counts-per-bucket; those *do* add across machines bucket-by-bucket, and you read the p99 off the summed histogram. Pooling the raw requests, as this script does for the "true" number, is the same thing at full resolution. This is why production telemetry stores mergeable structures like **HdrHistogram** or **t-digest** per machine and never averages percentiles across them. **The boundary condition.** Averaging per-group percentiles is exactly correct only when every group has an identical distribution *and* equal weight — the case where the groups are interchangeable and you gain nothing by splitting them. Any real heterogeneity (one slow server, one bad minute, uneven load) breaks it, and heterogeneity is the whole reason you were looking at per-machine numbers in the first place. ### Go deeper 1. **Flip the direction.** Predict first: make server 0 *low-volume* instead (e.g. 5,000 requests) while keeping it slow, and give the healthy servers more. Does the naive average now *over*state the true p99? Edit `PER_SERVER` per server and re-run. 2. **Add the histograms for real.** Replace the "pool every raw request" step with fixed-width buckets per server, sum the bucket counts across servers, and read p99 off the merged histogram. Confirm it matches the true pooled p99 — and think about how much less memory it takes than 1,000,000 floats. 3. **Average over time, not machines.** Recast the ten "servers" as ten consecutive one-minute windows of a single service, one of which had a GC pause. The same math indicts the common practice of averaging per-minute p99s into an hourly p99.