Your Own Write, Gone from the Replica

Write to the leader, immediately read a follower, and your own submission looks lost — a real Postgres standby returns count 0 for the comment you just posted, and reappears only after it catches up ~5 s later. You close the anomaly by routing reads-after-writes back to the leader.

Concept

Write to the leader, then read from a follower — the classic scale-out pattern, since data is read far more than written. But followers replicate asynchronously, so for a moment the follower hasn't seen your write yet. Read it in that window and your own submission looks lost — the comment you just posted isn't there. You will write to a real Postgres primary, immediately read a real streaming replica, watch your write come back as count 0, and see it reappear only after the replica catches up ~5 seconds later. Then you close the anomaly by routing reads-after-writes back to the leader.

Provenance

Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 6 — Replication, §"Reading your own writes":

if the user views the data shortly after making a write, the new data may not yet have reached the replica. To the user, it looks as though the data they submitted was lost.

The book names the anomaly and the guarantee that fixes it (read-your-writes / read-after-write consistency). Here you reproduce it on real streaming replication and measure the window.

Prerequisites

The surprise is about the gap between a leader's commit and a follower's apply. These help; each line notes where it shows up.

1. Leader-follower (single-leader) replication — writes go to one leader; followers copy its change log and serve reads. Reading a follower offloads the leader — the whole point.
2. Asynchronous replication — the leader commits and replies to the client without waiting for followers, so a follower is always slightly behind. That "behind" is the whole exercise.
3. Replication lag — the delay between a leader commit and a given follower reflecting it. Here we set it to 5 s (recovery_min_apply_delay) so it's large and obvious instead of a sub-millisecond race.
4. Read-your-writes consistency — the guarantee that a user always sees their own recent writes (others' can lag). The fix in Step 3.

Mental model

setup_replication.sh stands up a primary (leader) and a standby (follower) with real Postgres streaming replication. The standby is deliberately held 5 seconds behind (recovery_min_apply_delay = 5s) so the lag is easy to see. You write a comment to the primary and immediately read it back from both nodes; the follower hasn't applied it yet. The setup is scripted (it's just plumbing); the writes and reads in the Steps you run by hand.

Setup

cd study/ddia/ch06/code
bash setup_replication.sh     # ~30 s: primary + a fresh and a 5s-lagged standby

# handles for the two nodes:
#   primary (leader):        docker exec ddia-ch6-primary psql -U postgres -d study
#   standby (5s follower):   docker exec ddia-ch6-standby  psql -U postgres -d study

Do not read the output yet — make each prediction first.

# on the PRIMARY (leader):
docker exec ddia-ch6-primary psql -U postgres -d study -c \
  "INSERT INTO comments VALUES (1, 'my first comment');"

# then immediately, on the PRIMARY and the STANDBY:
docker exec ddia-ch6-primary psql -U postgres -d study -c "SELECT count(*) FROM comments WHERE id=1;"
docker exec ddia-ch6-standby psql -U postgres -d study -c "SELECT count(*) FROM comments WHERE id=1;"

# poll the standby until the row appears
until docker exec ddia-ch6-standby psql -U postgres -d study -tAc \
  "SELECT count(*) FROM comments WHERE id=1;" | grep -q 1; do sleep 0.2; done
docker exec ddia-ch6-standby psql -U postgres -d study -c "SELECT body FROM comments WHERE id=1;"

docker exec ddia-ch6-primary psql -U postgres -d study -c "SELECT body FROM comments WHERE id=1;"
#  ->  my first comment      (the leader always has it)

What you should see

• Immediately after the write: primary = 1, standby = 0 (your own write missing from the follower).
• The standby reflects it after ~5 s (the configured apply delay); the write was never lost.
• Reading the leader after a write always shows it — the read-your-writes fix.

Why

Single-leader replication scales reads by letting many followers serve queries, but it keeps writes correct by funneling them through one leader. To keep the leader fast, replication is asynchronous: the leader appends the change to its write-ahead log, commits, and answers the client immediately, then streams the log to followers, which apply it a little later. That "a little later" is replication lag, and during it a follower's state is an earlier version of the database. A read routed there sees the past — including, briefly, a past in which your just-submitted write doesn't exist yet. Nothing is broken and nothing is lost; you simply read a replica before it caught up. We forced the gap to 5 s with recovery_min_apply_delay, but under normal load it's just small enough to slip through in real traffic and confuse exactly the user who wrote the data.

The fix follows from naming the guarantee you actually need. You don't need every follower to be current — that would throw away the whole benefit of async replication. You need the narrow promise that the author of a write sees their own write: read-your-writes consistency. Implement it by routing reads-after-writes to the leader, or to a follower you've confirmed has replayed past the write's log position (Postgres exposes the position as an LSN), or by having the client remember the timestamp/LSN of its last write and only reading replicas that have reached it. Everyone else keeps hitting the cheap stale followers, which is fine — they never wrote the data, so they can't tell it's a few milliseconds old.

Go deeper

1. Read your writes by LSN. After the insert, capture pg_current_wal_lsn() on the primary; poll the standby's pg_last_wal_replay_lsn() until it passes that LSN, then read the standby. Predict whether you can get a fresh read from the follower this way (you can) — the without-hitting-the-leader version of the fix.
2. Sync vs async latency. Add synchronous_standby_names so the primary waits for the standby to confirm each commit. Predict how much a single INSERT's latency grows — the cost you pay to erase the lag window.
3. Two windows. Lower recovery_min_apply_delay to 500ms and re-run. The anomaly still happens, just harder to catch by hand — which is exactly why it slips into production unnoticed.

Sources: DDIA 2e, Ch. 6, §"Reading your own writes", §"Solutions for Replication Lag" · the PostgreSQL streaming replication docs (pg_stat_replication, recovery_min_apply_delay).