A loan model discriminates against a protected group, so you apply the obvious fix — delete the protected column. With the attribute the disparate-impact ratio is 0.29; drop it and it barely moves to 0.44, still failing the 0.80 four-fifths rule, because the model reconstructs the trait from a correlated proxy (a second model recovers it at AUC 0.96). Only dropping the proxy too reaches 1.03. "Machine learning is like money laundering for bias."
A loan model is discriminating against a protected group. The obvious fix is fairness through unawareness: don't feed the model the protected attribute (race), and its decisions become fair. This exercise falsifies that. On a seeded synthetic dataset where the protected trait A is strongly correlated with a neutral-looking proxy (a "neighborhood" / postal code) and the historical labels carry bias against A=1, you train the same classifier three ways and audit each with the disparate-impact ratio (four-fifths rule: below 0.80 is a legal red flag). With A the ratio is 0.29; drop A and it barely moves to 0.44 — still failing — because the model reconstructs A from the proxy (a second model recovers A from the remaining features at AUC 0.96). Only when you drop the proxy too does the ratio jump to 1.03. Removing the column removes the name, not the information. As the book puts it, "machine learning is like money laundering for bias."
Designing Data-Intensive Applications, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 14 — Doing the Right Thing, §"Bias and Discrimination":
in racially segregated neighborhoods, a person's postal code or even their IP address is a strong predictor of race … machine learning is like money laundering for bias.
This is the ethics chapter's one crisp, mechanical claim — and it is testable. Everything else in the chapter is argument. Here you make the "just drop the column" prediction, run it, and watch a correlated proxy carry the trait straight back in. (This is a lab about why not to ship such a system; the biased model is the object of study, never the goal.)
The surprise is that dropping a column doesn't drop the information in it. These help; each line notes where it shows up.
A from the other features.uv with scikit-learn + numpy; no Docker; seeded (numpy default_rng(42), every model random_state=42). 20,000 synthetic applicants, 70/30 train/test split (stratified, seed 42).
One synthetic loan dataset, three models, one audit metric.
A (A=1 is the historically disadvantaged group). A proxy feature, neighborhood (think postal code), drawn so the two groups live in different places — it strongly encodes A without ever being labelled A. Three legitimate creditworthiness features (income, credit score, employment years) drawn independently of A, so the groups are equally qualified on the merits. The label is historically biased: approval = merit − a penalty applied to A=1. The bias lives entirely in A; the proxy is only correlated with it.random_state=42, trained on the biased labels):
A.A ("fairness through unawareness").A and without the proxy.A=1) ÷ selection-rate(A=0), computed on the held-out test set using each applicant's true group (even for M2/M3, which never see A as a feature — you still know who's who when auditing). Below 0.80 fails the four-fifths rule.The whole thing is one script, code/proxy_bias.py.
cd study/ddia/ch14/code
uv run --python 3.12 --with scikit-learn --with numpy python3 proxy_bias.pyDo not read the output yet — make each prediction first. (First run downloads scikit-learn into a throwaway uv env; a few seconds thereafter.)
Predict. The groups are equally qualified, but the historical labels encode a penalty on A=1. Roughly what disparate-impact ratio do the labels themselves carry — and does it fail the 0.80 rule?
0.42 — the labels are the problem. Equally-qualified A=1 applicants were approved at 29% vs 69% for A=0. Any model that learns to reproduce these labels inherits the bias. That's the starting point; the question is what "unawareness" does to it.
Predict. Train the classifier on every feature, A included. What disparate-impact ratio does its test-set decisions show?
0.29 — the model faithfully learned the discrimination (and sharpened it: it uses A directly, so it splits the two groups even harder than the raw labels). This is the biased baseline we're trying to fix.
Predict. Now remove the A column entirely and retrain — the classic fix. The model literally cannot see race. What does the ratio become — does it climb to ≈ 1.0 (fair)?
0.44 — it barely moved, and still fails 0.80. Blinding the model to race recovered almost none of the fairness. The decisions are still overwhelmingly against A=1, even though A is gone from the training data. Something else in the features is standing in for it.
Predict. If M2 is still biased without A, the trait must still be present in the other columns. Train a second model to predict A from the non-A features. How well can it do — near 0.50 (chance) or much higher?
AUC 0.96 — the protected attribute is trivially reconstructable, and the proxy alone accounts for all of it (0.96 either way). "Dropping A" removed the label on the column, not the information: neighborhood is a near-perfect stand-in, so M2 relearns A and reproduces the bias. This is the money-laundering step — the forbidden trait is washed through a permitted feature.
Predict. So the proxy is the leak. Drop A and neighborhood, leaving only the merit features (which are independent of A). Now what does the ratio become?
1.03 — fair, and it jumped there only when the proxy left. With no feature that encodes group membership, the model can only act on genuine merit, which is equal across groups, so the selection rates equalize. The contrast between Step 3 (0.44, proxy present) and Step 5 (1.03, proxy gone) is the proof that the proxy — not the labelled A column — was carrying the bias.
A): 0.29 — faithfully learns the bias.A): 0.44 — barely moves; still fails the 0.80 rule.A from the remaining features: AUC 0.96 (the proxy alone).A and the proxy): 1.03 — fair.Start from what a model is doing: minimizing error against the training labels. Those labels are biased — equally-qualified A=1 applicants were approved far less often. So any model good at fitting them will reproduce the bias, provided it can tell the groups apart. The only question is whether the features let it.
"Fairness through unawareness" bets that deleting the A column removes the model's ability to tell the groups apart. That bet is false whenever a proxy exists — a permitted feature statistically entangled with the forbidden one. Here neighborhood was generated so the two groups live in different places, which is exactly the real-world fact the book cites: under residential segregation, postal code predicts race. Information-theoretically, A and neighborhood share mutual information; deleting A leaves that information sitting in neighborhood, fully available to the model. The recovery model quantifies it: A is reconstructable from the proxy at AUC 0.96, essentially the whole trait. A model "free of race" that can recover race at 0.96 is free of race in name only.
Now the arithmetic of the three ratios. M1 reads A directly and drives the ratio to 0.29. M2 loses the A column but keeps a 0.96-fidelity copy of it in neighborhood, so it reconstructs group membership and reproduces almost the same bias — 0.44, a rounding error away from unfair, still under 0.80. It moved slightly only because the proxy is a 0.96 copy, not a perfect 1.0 one; the small residual noise is the only thing "unawareness" bought. M3 finally deletes the proxy, and now no remaining feature encodes the group (the merit features were drawn independently of A), so the model can only act on true creditworthiness, which is equal across groups — the ratio equalizes to 1.03. The lesson is that the ratio tracks how much group information reaches the model, and dropping a labelled column doesn't reduce that if a proxy re-supplies it.
A's removal alone. You can see the boundary inside this very exercise — M3's features are proxy-free, and there unawareness succeeds. The catch is that in real high-dimensional data such a proxy almost always exists (postal code, IP, name, shopping history, device), and often you can't even enumerate the proxies to remove them — which is why practitioners audit outcomes (disparate impact) rather than trusting inputs (which columns were dropped). "We don't collect race" is not a fairness guarantee.
code/proxy_bias.py shrink the group separation in neighborhood (raise scale toward the within-group spread). Predict the recovery AUC dropping toward 0.50 and M2's ratio climbing toward M3's — watch unawareness start to work exactly as the proxy's mutual information with A fades. That sweep is the boundary condition.Sources: DDIA 2e, Ch. 14, §"Bias and Discrimination" · Barocas & Selbst, "Big Data's Disparate Impact" (2016) · Dwork et al., "Fairness Through Awareness" (2012) · Chouldechova (2017); Kleinberg, Mullainathan & Raghavan (2016).