# DDIA Ch. 7: a timestamp partition key funnels 100% of writes into one shard ## Concept Key-range sharding gives each shard a **contiguous** slice of the key space — the property that makes a range scan cheap, because adjacent keys live together. But choose a **monotonically increasing** partition key (a timestamp) and that same ordering turns against you: the maximum key always lives in the *last* shard, so a write-as-you-go workload funnels **100% of writes into one shard** while the other 11 sit idle. A self-inflicted write hot spot. The book's fix is to prefix the key with something high-cardinality — the **sensor id** — so the write point scatters across the sensor-id range and load spreads evenly (busiest shard drops to **1.024×** the mean). The documented cost lands immediately: a time-range query *across all sensors*, one contiguous scan under the timestamp key, now touches **all 12 shards** — one scan per sensor. Same 100,000 readings, same 12 range shards; the only thing that changes is which field the key sorts by. ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 7 — *Sharding*, §"Sharding by Key Range" (the sensor-network example): > The downside of key-range sharding is that it can lead to hot spots if the key > [...] is a timestamp, [...] because all writes end up going to the same > partition (the one for the current day) [...]. To avoid this problem [...] you > could prefix each timestamp with the sensor name so that the [sharding] is > first by sensor name and then by time. The book warns that the fix has a price: "if you want to fetch the values of multiple sensors within a time range, you now need to perform a separate range query for each sensor name." This exercise makes both halves — the hot spot and its cost — into numbers you can predict against. ## Prerequisites The surprise is about how *key order* interacts with *write order*. Each line notes where it shows up. 1. **Key-range vs. hash sharding** — key-range gives each shard a contiguous span of keys (cheap range scans, risky hot spots); hashing scatters keys to spread load (even writes, but no range scans). This exercise is entirely inside the key-range world; the hot spot is the price of contiguity. It shows up as the `shard_by_timestamp` / `shard_by_sensor_key` range routers. 2. **Sequential vs. random key access** — a monotonic key (timestamp) writes to one ever-advancing point; a high-cardinality key (sensor id) writes to many points at once. It shows up as scenario A's single busy shard vs. scenario B's twelve. 3. **Data locality** — placing related rows physically together makes reading them cheap; the cross-sensor time-range query is fast exactly when the matching keys are contiguous. It shows up as "shards touched = 1" vs. "= 12". ### Where to learn the prerequisites - **Key-range vs. hash sharding (#1):** DDIA §"Sharding by Key Range" and §"Sharding by Hash of Key" — the two schemes and the trade-off between range scans and even load. - **Sequential-key hot spots (#2):** the AWS/Google "avoid monotonically increasing keys" guidance for DynamoDB partition keys and Bigtable/Spanner row keys — the same anti-pattern in production docs. - **Data locality (#3):** DDIA §"Sharding by Key Range" on why keeping a range physically ordered lets a scan read it in one sweep. If only one is new, make it #2 — a sequential partition key is the entire cause of the hot spot. ## Environment Deterministic, so the counts reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.15.0a8, standard library only — no Docker, no deps - The only randomness is sensor assignment, seeded `random.Random(42)` - Fixed workload: 100,000 readings, 1,000 sensors, 12 range shards, timestamps strictly increasing within the current month ## Mental model Two **partition-key choices** over the *identical* stream of 100,000 readings. The shards are range-based both times (each owns a contiguous slice); only the key differs. - **Scheme A — key = `timestamp`.** Shards partition *time*: shard `m` owns the contiguous range `[m·MONTH, (m+1)·MONTH)`. Because every reading arrives with the current timestamp, and the current timestamp is always the maximum key seen so far, every write lands in the newest shard. Cheap time-range scans; a guaranteed write hot spot. - **Scheme B — key = `f"{sensor}#{timestamp}"`.** The key sorts by *sensor* first, so shards partition the *sensor-id* space: shard `j` owns a contiguous slice of the 1,000 sensor ids. The write point is now wherever the sensor lands, and sensors are drawn across the whole id range — so writes spread. Even load; but a cross-sensor time range is no longer contiguous. Both schemes route with a range boundary lookup (integer division for time, a `bisect` over sensor-id boundaries for the string key). The whole thing is one script, `code/key_range_hot_spot.py`. ## Setup ``` cd study/ddia/ch07 python3 code/key_range_hot_spot.py ``` Do **not** read the output yet — make each prediction first. ## Steps ### Step 1 — timestamp partition key: where do the writes land? **Predict.** 100,000 readings stream in with monotonically increasing timestamps, all inside the current month, across 12 shards that each own one month of time. How many of the 12 shards receive writes, and what share does the busiest hold? **Observe.** ``` per-shard write counts: shard 0 : 0 [........................................] shard 1 : 0 [........................................] shard 2 : 0 [........................................] shard 3 : 0 [........................................] shard 4 : 0 [........................................] shard 5 : 0 [........................................] shard 6 : 0 [........................................] shard 7 : 0 [........................................] shard 8 : 0 [........................................] shard 9 : 0 [........................................] shard 10 : 0 [........................................] shard 11 : 100,000 [########################################] busiest shard holds 100,000 of 100,000 writes = 100.0% 11 of 12 shards are completely idle -- a self-inflicted write HOT SPOT. ``` **One shard holds everything.** Shard 11 owns the current month, the current month holds the maximum key, and every write carries a fresh maximum — so 100.0% of writes hit shard 11 and the other 11 shards never wake up. Buying 12 shards bought exactly 1× the throughput of a single machine. ### Step 2 — sensor-prefixed key: does the load spread? **Predict.** Same 100,000 readings, but the key is now `f"{sensor}#{timestamp}"` and the shards partition the 1,000-sensor id space. Roughly how evenly do the writes spread — and how far is the busiest shard from the mean of 8,333? **Observe.** ``` per-shard write counts: shard 0 : 8,292 [###.....................................] shard 1 : 8,431 [###.....................................] shard 2 : 8,133 [###.....................................] shard 3 : 8,268 [###.....................................] shard 4 : 8,465 [###.....................................] shard 5 : 8,405 [###.....................................] shard 6 : 8,129 [###.....................................] shard 7 : 8,537 [###.....................................] shard 8 : 8,365 [###.....................................] shard 9 : 8,452 [###.....................................] shard 10 : 8,283 [###.....................................] shard 11 : 8,240 [###.....................................] max 8,537 min 8,129 mean 8,333 imbalance: busiest shard = 1.024x the mean (vs 12.0x under scenario A). writes now SPREAD across all 12 shards. ``` **All 12 shards busy, none more than 2.4% above the mean.** Prefixing with the sensor id moved the write point from "the single newest key" to "wherever this sensor sorts," and 1,000 sensors spread across the id range keep every shard fed. The hot spot is gone. ### Step 3 — the cost: a time-range query across all sensors **Predict.** Now ask a read: *all readings in a 20,001-second window, across all sensors.* Under the timestamp key (scheme A) and under the sensor-prefixed key (scheme B), how many of the 12 shards must the query touch? **Observe.** ``` window [t0,t1] = [28,552,000 .. 28,572,000] -> 20,001 readings match scheme A (key = timestamp) : matching keys are CONTIGUOUS in time -> shards touched = 1 [11] scheme B (key = sensor#timestamp): matching keys are SCATTERED by sensor -> shards touched = 12 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] Spreading the writes cost the query 12x the shards to scan: one contiguous range scan became 12 scans (one per shard). ``` **Scheme A touches 1 shard; scheme B touches all 12.** Under the timestamp key the matching readings are contiguous — one range on one shard. Under the sensor-prefixed key they are sorted by sensor, so the same 20,001 readings are smeared across every shard: the query becomes a separate range scan per sensor. The fix for the write hot spot *is* the cause of the read fan-out. ## What you should see - Timestamp key: the busiest shard holds **100.0%** of writes; **11 of 12** shards idle. - Sensor-prefixed key: writes spread to all **12** shards, busiest **1.024×** the mean (max 8,537, min 8,129, mean 8,333). - The cross-sensor time-range query touches **1** shard under the timestamp key and **12** under the sensor-prefixed key — a **12×** read fan-out, the price of the even writes. ## Why Key-range shards are contiguous *by construction*: shard `m` owns keys in `[boundary_m, boundary_{m+1})`, and that is exactly what lets a range scan read a span of keys as one sequential sweep. The trouble is that "contiguous by key order" and "hot on sequential writes" are the same coin. A timestamp key is monotonic — every new reading's key is greater than every key written before it — so the write point is pinned to the top of the key space, and the top of the key space is, by the range-partition definition, a single shard: the last one. There is no arithmetic that spreads a stream of ever-increasing keys across contiguous ranges, because "ever-increasing" means "always in the same top range." The 100.0% is not bad luck; it is forced by the two design choices meeting. Prefixing the key with the sensor id changes what the *first* sort field is. Now the key is `sensor#timestamp`, so the shard is chosen by sensor, and the 1,000 sensor ids — drawn across the whole id range — scatter the write point over all 12 ranges. The write hot spot dissolves because the leading bytes of the key are high-cardinality and roughly uniform, which is the general recipe: **a key-range write hot spot is cured by making the key's high-order part high-cardinality.** But you paid for it in the currency the range scan was denominated in. A "same time window, all sensors" query selects a *horizontal* slice of `(sensor, time)` space. Under `timestamp` keys that slice is one contiguous run (all keys with `t ∈ [t0,t1]` are adjacent) → one shard. Under `sensor#timestamp` keys that slice is *vertical* to the sort order — it crosses every sensor prefix — so its keys are spread over all 12 shards, one range scan each. The same locality you destroyed to spread the writes was the locality that made the cross-sensor scan cheap. **The boundary — the hot spot is specific to sequential keys.** Nothing here happens if the partition key is *not* monotonic. Hash the timestamp, use a random UUID, or use any key whose high-order bits are already high-cardinality, and key-range (or hash) sharding spreads the writes on its own — there is no newest shard to funnel into. That is the precise condition: the funnel exists only when successive writes carry successive keys. Equally, the read fan-out is specific to queries whose predicate cuts *across* the leading sort field; a query for one sensor's time range under scheme B is still one contiguous scan on one shard. The tension is not "range-sharding is bad" — it is that **the field you sort by first is simultaneously the field your writes spread over and the field your range scans stay cheap over**, and a timestamp cannot be both the thing that spreads writes and the thing that keeps time-range reads local. You choose which query gets the contiguity. ### Go deeper 1. **Move the window.** Change `[t0,t1]` to span two calendar months and watch scheme A jump from 1 shard to 2 — the contiguity is only as tight as the window, and a year-long query would touch all 12 even under the timestamp key. 2. **Break the monotonicity.** Replace the timestamp key with `hash(timestamp)` (or prepend a random byte) and re-measure scenario A — the 100.0% hot spot should collapse toward the even spread of scenario B, showing the funnel was a property of *sequential keys*, not of range sharding. 3. **Count the wasted parallelism.** Scheme B touches all 12 shards for the cross-sensor query but each does 1/12th of the work in parallel; scheme A touches 1 shard doing all the work serially. Add per-shard row counts to the query and decide which you'd actually rather have — the answer depends on whether your bottleneck is the hot write shard or the read latency.