Ten servers each report their own p99. Average those numbers into one "fleet p99" and it lands 21% below the true pooled p99 — with 78% of the real tail coming from a single degraded server that handles only 10% of traffic. You can't average percentiles.
Every machine behind a load balancer reports its own p99. The obvious way to get a single "fleet p99" is to average those per-machine numbers. That average is mathematically meaningless — it is not the p99 of anything. This exercise builds a 10-server fleet with one degraded machine, averages the per-server p99s, and compares that to the true p99 of all requests pooled together. You will predict "close enough" and watch the averaged number land 21% below the real tail — with 78% of that tail coming from a single server that handles only 10% of traffic.
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 2 — Defining Nonfunctional Requirements, §"Computing Percentiles":
When you have several HTTP calls … you need to be careful about how you aggregate response times … averaging percentiles is meaningless. The correct way of aggregating response time data is to add the histograms.
The book states the rule. Here you produce the error the rule warns against and measure how big it is.
The surprise is a fact about what a percentile is and what it throws away. These help; each line notes where it shows up.
p99(A ∪ B) ≠ mean(p99(A), p99(B)) in general. The core claim; the whole run is a demonstration of the inequality.random.Random(42), so your run reproduces these exact figures. A different arch/version shifts the digits slightly; the shape holds.
Ten servers sit behind a load balancer, each handling the same number of requests (100,000; 1,000,000 total). Nine are healthy — the same lognormal as the other exercises, median 10 ms, p99 ≈ 100 ms. One (server 0) is degraded: 4× slower median (40 ms), same spread, so its whole latency curve is shifted up. Each server computes its own p99. We compare two "fleet p99" numbers:
The whole exercise is one script, code/averaging_percentiles.py (~90 lines of stdlib). Read it once, then run it and check each prediction below.
cd study/ddia/code
# no dependencies; just Python 3
python3 averaging_percentiles.py # ~3 s: simulates 1,000,000 requestsDo not read the output yet — make each prediction first.
Predict. Nine healthy servers (p99 ≈ 100 ms) and one degraded server (4× slower). Before looking: roughly what is the degraded server's own p99?
python3 averaging_percentiles.py # read the per-server blockNine servers cluster at ~100 ms; the sick one is at 393 ms. Nothing surprising yet — this is exactly the data a per-machine dashboard shows.
Now collapse those ten numbers into one "fleet p99" the obvious way: average them. Then compare to the true p99 of every request pooled together.
Predict. How close is mean(per-server p99s) to the true pooled p99? Write down a guess for both. (Most people expect them within a few percent — "it's an average of p99s, it should be about the p99.")
The averaged number is 129 ms; the real p99 is 163 ms. The average understates the tail your users actually feel by 21% — and it is not even a predictable direction (see Go deeper). The two numbers are answers to different questions.
Look at the last line (no new run needed).
Predict. The degraded server handles 1 in 10 requests. Of the fleet's slowest 1% overall, what share came from it?
78%. The one sick server, though only a tenth of traffic, produced more than three-quarters of the slowest 1% of all requests. Averaging the ten p99s divides its spike by ten and throws that away — which is precisely why the naive number misses.
A percentile is a lossy summary: it reports one point of a distribution and discards the rest of its shape. Server 0's "p99 = 393 ms" tells you one number and hides that server 0 also emitted tens of thousands of requests in the 150–400 ms range. The fleet's true p99 is decided by the 10,000 slowest requests out of 1,000,000 — and 78% of those are server 0's. When you average the ten per-server p99s, you compress server 0's entire heavy upper tail into one value and then divide it by ten. You have thrown away exactly the information the pooled p99 depends on, so the average cannot reconstruct it. In symbols: p99(A ∪ B) ≠ mean(p99(A), p99(B)), because a percentile of a union is not a function of the percentiles of the parts.
The direction of the error isn't even fixed. Here the sick server has average volume, so its many slow requests flood the pooled tail and the naive average understates the truth. Give it low volume instead — a small, very slow node — and its huge p99 still enters the average at weight 1/10 while barely touching the pool, so the naive number would overstate the real p99. You cannot predict even the sign of the error from the per-server p99s alone. That is what "meaningless" means: the information required simply isn't in the inputs.
The correct aggregation is the one the book names: add the histograms. Keep each machine's full latency distribution as counts-per-bucket; those do add across machines bucket-by-bucket, and you read the p99 off the summed histogram. Pooling the raw requests, as this script does for the "true" number, is the same thing at full resolution. This is why production telemetry stores mergeable structures like HdrHistogram or t-digest per machine and never averages percentiles across them.
PER_SERVER per server and re-run.Sources: DDIA 2e, Ch. 2, §"Computing Percentiles" · on mergeable percentile structures, see HdrHistogram (Gil Tene) and t-digest (Ted Dunning).