# DDIA Ch. 14: dropping "race" doesn't debias the model — a proxy relearns it ## Concept A loan model is discriminating against a protected group. The obvious fix is **fairness through unawareness**: don't feed the model the protected attribute (race), and its decisions become fair. This exercise falsifies that. On a seeded synthetic dataset where the protected trait `A` is strongly correlated with a neutral-looking **proxy** (a "neighborhood" / postal code) and the historical labels carry bias against `A=1`, you train the same classifier three ways and audit each with the **disparate-impact ratio** (four-fifths rule: below **0.80** is a legal red flag). With `A` the ratio is **0.29**; drop `A` and it barely moves to **0.44** — still failing — because the model reconstructs `A` from the proxy (a second model recovers `A` from the remaining features at **AUC 0.96**). Only when you drop the proxy *too* does the ratio jump to **1.03**. Removing the column removes the *name*, not the *information*. As the book puts it, "machine learning is like money laundering for bias." ## Provenance *Designing Data-Intensive Applications*, 2nd ed. (Kleppmann & Riccomini, 2025), Chapter 14 — *Doing the Right Thing*, §"Bias and Discrimination": > in racially segregated neighborhoods, a person's postal code or even their IP > address is a strong predictor of race … machine learning is like money > laundering for bias. This is the ethics chapter's one crisp, mechanical claim — and it is testable. Everything else in the chapter is argument. Here you make the "just drop the column" prediction, run it, and watch a correlated proxy carry the trait straight back in. (This is a lab about why *not* to ship such a system; the biased model is the object of study, never the goal.) ## Prerequisites The surprise is that dropping a column doesn't drop the information in it. These help; each line notes where it shows up. 1. **Disparate impact & the four-fifths (0.80) rule** — a fairness audit: the ratio of selection rates between the disadvantaged and advantaged group; U.S. guidelines flag a ratio below 0.80. It shows up as the number we print for every model. 2. **Proxy variables / redundant encoding** — a permitted feature that is statistically entangled with a forbidden one, so it silently re-encodes it (postal code ↔ race under residential segregation). **This is the concept that carries the whole result.** 3. **Fairness through unawareness** — the belief that omitting the protected attribute makes a model fair. It shows up as model **M2**, and its failure is the aha. 4. **A classifier's AUC** — area under the ROC curve: the probability the model ranks a random positive above a random negative; 0.50 is chance, 1.0 perfect. It shows up as the score of the recovery model that predicts `A` from the other features. ### Where to learn them - **Disparate impact / four-fifths (#1):** the EEOC Uniform Guidelines on Employee Selection; Feldman et al., "Certifying and Removing Disparate Impact" (2015). - **Proxies / redundant encoding (#2):** Barocas & Selbst, "Big Data's Disparate Impact" (2016); the free online book *Fairness and Machine Learning* (Barocas, Hardt, Narayanan), fairmlbook.org, ch. "Classification." - **Fairness through unawareness (#3):** Dwork et al., "Fairness Through Awareness" (2012) — the paper that names and rejects the unawareness approach. - **AUC (#4):** Google's Machine Learning Crash Course, "Classification: ROC and AUC." If only one is new, make it #2 — a proxy that is predictive of the protected attribute is the entire mechanism; without it, dropping the column *would* work. ## Environment Seeded and deterministic, so the metrics reproduce exactly: - macOS 26.5.2 (Darwin 25.5.0), arm64 - Python 3.12 via `uv` with scikit-learn + numpy; no Docker - Seeded: numpy `default_rng(42)`, every model `random_state=42` - 20,000 synthetic applicants, 70/30 train/test split (stratified, seed 42) ## Mental model One synthetic loan dataset, three models, one audit metric. - **The data.** A protected binary attribute `A` (`A=1` is the historically disadvantaged group). A **proxy** feature, `neighborhood` (think postal code), drawn so the two groups live in different places — it strongly encodes `A` without ever being *labelled* `A`. Three **legitimate** creditworthiness features (income, credit score, employment years) drawn *independently* of `A`, so the groups are **equally qualified on the merits**. The **label** is historically biased: approval = merit − a penalty applied to `A=1`. The bias lives entirely in `A`; the proxy is only *correlated* with it. - **The models** (same RandomForest, `random_state=42`, trained on the biased labels): - **M1** — trained **with** `A`. - **M2** — trained **without** `A` ("fairness through unawareness"). - **M3** — trained **without** `A` *and* without the proxy. - **The audit.** Disparate-impact ratio = selection-rate(`A=1`) ÷ selection-rate(`A=0`), computed on the held-out test set using each applicant's *true* group (even for M2/M3, which never see `A` as a feature — you still know who's who when auditing). Below 0.80 fails the four-fifths rule. The whole thing is one script, `code/proxy_bias.py`. ## Setup ``` cd study/ddia/ch14/code uv run --python 3.12 --with scikit-learn --with numpy python3 proxy_bias.py ``` Do **not** read the output yet — make each prediction first. (First run downloads scikit-learn into a throwaway `uv` env; a few seconds thereafter.) ## Steps ### Step 1 — the biased history **Predict.** The groups are equally qualified, but the *historical labels* encode a penalty on `A=1`. Roughly what disparate-impact ratio do the labels themselves carry — and does it fail the 0.80 rule? **Observe.** ``` Disparate impact in the HISTORICAL LABELS themselves: approval rate A=1 (disadvantaged) = 29.0% approval rate A=0 (advantaged) = 69.4% disparate-impact ratio = 0.42 (four-fifths rule: below 0.80 is a red flag) ``` **0.42 — the labels are the problem.** Equally-qualified `A=1` applicants were approved at 29% vs 69% for `A=0`. Any model that learns to reproduce these labels inherits the bias. That's the starting point; the question is what "unawareness" does to it. ### Step 2 — M1: train WITH the protected attribute **Predict.** Train the classifier on every feature, `A` included. What disparate-impact ratio does its test-set decisions show? **Observe.** ``` M1 -- train WITH the protected attribute A ==================================================================== selection rate A=1 = 21.5% A=0 = 75.0% disparate-impact ratio = 0.29 ``` **0.29 — the model faithfully learned the discrimination** (and sharpened it: it uses `A` directly, so it splits the two groups even harder than the raw labels). This is the biased baseline we're trying to fix. ### Step 3 — M2: drop `A` (fairness through unawareness) **Predict.** Now remove the `A` column entirely and retrain — the classic fix. The model literally cannot see race. What does the ratio become — does it climb to ≈ 1.0 (fair)? **Observe.** ``` M2 -- train WITHOUT A ('fairness through unawareness': drop race) ==================================================================== selection rate A=1 = 29.4% A=0 = 66.4% disparate-impact ratio = 0.44 <-- barely moved; still unfair ``` **0.44 — it barely moved, and still fails 0.80.** Blinding the model to race recovered almost none of the fairness. The decisions are still overwhelmingly against `A=1`, even though `A` is gone from the training data. Something else in the features is standing in for it. ### Step 4 — the mechanism: recover `A` from the remaining features **Predict.** If M2 is still biased without `A`, the trait must still be present in the other columns. Train a *second* model to predict `A` from the non-`A` features. How well can it do — near 0.50 (chance) or much higher? **Observe.** ``` MECHANISM CHECK -- recover A from the non-protected features ==================================================================== predict A from ALL non-A features : AUC = 0.96 predict A from the PROXY alone : AUC = 0.96 (0.50 = coin flip. Well above 0.50 => the trait is still in there.) ``` **AUC 0.96 — the protected attribute is trivially reconstructable**, and the proxy alone accounts for all of it (0.96 either way). "Dropping `A`" removed the *label* on the column, not the *information*: `neighborhood` is a near-perfect stand-in, so M2 relearns `A` and reproduces the bias. This is the money-laundering step — the forbidden trait is washed through a permitted feature. ### Step 5 — M3: drop the proxy too **Predict.** So the proxy is the leak. Drop `A` **and** `neighborhood`, leaving only the merit features (which are independent of `A`). Now what does the ratio become? **Observe.** ``` M3 -- train WITHOUT A AND WITHOUT the proxy (neighborhood) ==================================================================== selection rate A=1 = 49.3% A=0 = 48.0% disparate-impact ratio = 1.03 <-- now above 0.80 ``` **1.03 — fair, and it jumped there only when the proxy left.** With no feature that encodes group membership, the model can only act on genuine merit, which is equal across groups, so the selection rates equalize. The contrast between Step 3 (0.44, proxy present) and Step 5 (1.03, proxy gone) is the proof that the *proxy* — not the labelled `A` column — was carrying the bias. ## What you should see - The historical labels carry a disparate-impact ratio of **0.42** (equally qualified groups, unequal approvals). - **M1** (with `A`): **0.29** — faithfully learns the bias. - **M2** (without `A`): **0.44** — barely moves; still fails the 0.80 rule. - Recovering `A` from the remaining features: **AUC 0.96** (the proxy alone). - **M3** (without `A` *and* the proxy): **1.03** — fair. ## Why Start from what a model is doing: minimizing error against the training labels. Those labels are biased — equally-qualified `A=1` applicants were approved far less often. So *any* model good at fitting them will reproduce the bias, provided it can tell the groups apart. The only question is whether the features let it. "Fairness through unawareness" bets that deleting the `A` column removes the model's ability to tell the groups apart. That bet is false whenever a *proxy* exists — a permitted feature statistically entangled with the forbidden one. Here `neighborhood` was generated so the two groups live in different places, which is exactly the real-world fact the book cites: under residential segregation, postal code predicts race. Information-theoretically, `A` and `neighborhood` share **mutual information**; deleting `A` leaves that information sitting in `neighborhood`, fully available to the model. The recovery model quantifies it: `A` is reconstructable from the proxy at **AUC 0.96**, essentially the whole trait. A model "free of race" that can recover race at 0.96 is free of race in name only. Now the arithmetic of the three ratios. **M1** reads `A` directly and drives the ratio to **0.29**. **M2** loses the `A` column but keeps a 0.96-fidelity copy of it in `neighborhood`, so it reconstructs group membership and reproduces almost the same bias — **0.44**, a rounding error away from unfair, still under 0.80. It moved *slightly* only because the proxy is a 0.96 copy, not a perfect 1.0 one; the small residual noise is the *only* thing "unawareness" bought. **M3** finally deletes the proxy, and now *no* remaining feature encodes the group (the merit features were drawn independently of `A`), so the model can only act on true creditworthiness, which is equal across groups — the ratio equalizes to **1.03**. The lesson is that the ratio tracks *how much group information reaches the model*, and dropping a labelled column doesn't reduce that if a proxy re-supplies it. **The boundary — the failure needs a real proxy.** If *no* remaining feature correlated with the protected attribute, dropping the column *would* work: with mutual information near zero, the model couldn't reconstruct the group, and the ratio would move to fair on `A`'s removal alone. You can see the boundary inside this very exercise — M3's features are proxy-free, and there unawareness *succeeds*. The catch is that in real high-dimensional data such a proxy almost always exists (postal code, IP, name, shopping history, device), and often you can't even enumerate the proxies to remove them — which is why practitioners audit *outcomes* (disparate impact) rather than trusting *inputs* (which columns were dropped). "We don't collect race" is not a fairness guarantee. ### Go deeper 1. **Weaken the proxy.** In `code/proxy_bias.py` shrink the group separation in `neighborhood` (raise `scale` toward the within-group spread). Predict the recovery AUC dropping toward 0.50 and M2's ratio climbing toward M3's — watch unawareness start to work exactly as the proxy's mutual information with `A` fades. That sweep *is* the boundary condition. 2. **Enforce fairness instead of hiding inputs.** Instead of deleting columns, post-process M1's scores to equalize selection rates (or use a fairness-constrained learner). This is the actual fix — an *outcome* constraint, not an *input* deletion. Read up on equalized odds and adversarial debiasing. 3. **Meet the impossibility results.** Try to make M1 satisfy *both* equal selection rates *and* equal error rates across groups at once. Discover why, when base rates differ, Chouldechova (2017) and Kleinberg et al. (2016) prove you generally can't — the fairness metrics themselves conflict.