Natural experiments

The idea of a natural experiment

In the previous guides we learned that the gold standard for proving causation is the randomized controlled trial: you flip a coin to decide who gets the treatment, so the two groups differ only by luck, and any difference that shows up afterward must have been caused by the treatment. But very often we simply cannot randomize. It may be unethical, illegal, far too expensive, or flatly impossible — you cannot randomly assign people to smoke, to drop out of school, or to be born during a recession.

When we give up and just compare the people who happened to get a treatment against those who didn't, we fall straight back into the trap of the last two guides: confounding. To recap, a confounder is a common cause of both the treatment and the outcome, and it manufactures a fake, spurious association (correlation is not causation). Classic example: people who use a fitness app are healthier — but perhaps health-conscious people both download the app and exercise anyway, so the app gets credit for something it never caused.

Why does this rescue us? If the thing that decided who got treated is unrelated to the confounders, then comparing treated against untreated is once again a fair comparison — and the difference we see can again be read as a causal effect. The entire craft of quasi-experimental design (quasi-experimental means: it looks like an experiment, but the assignment wasn't deliberately randomized) is finding a source of variation in the treatment that is plausibly disconnected from everything else that affects the outcome.

Three of the most useful natural-experiment designs are instrumental variables, difference-in-differences, and regression discontinuity. Each finds a different kind of accidental randomization hiding in the world. We'll meet them one at a time, each with real numbers, and then come back to the assumptions every one of them quietly relies on.

Instrumental variables: a nudge from outside

Suppose we want to know: does using our new budgeting feature cause people to save more money? The naive comparison is hopeless — motivated savers both turn on the feature and save more, so the feature looks great for reasons that have nothing to do with it. And we can't force anyone to use it. But here's a trick we can pull: we randomly email half of our users an encouragement to try the feature. That email is our instrument.

An instrumental variable (often just an instrument), call it Z, is a variable that (1) nudges the treatment X up or down, (2) has no other route to the outcome Y except through X, and (3) is itself as-good-as-random. A randomly-sent email fits beautifully: it bumps feature adoption, it shouldn't move anyone's savings except by getting them to use the feature, and because we randomized who received it, it's unrelated to how motivated each person already was.

Now the numbers. Among users we emailed, 40% adopted the feature and they saved an average of $312 per month. Among users we did not email, only 20% adopted, and they saved $300 per month. So the email did two things: it raised adoption by 20 percentage points (statisticians call this the first stage), and it raised average savings by $12 (the reduced form). But remember — the email itself doesn't move money. It can only have raised savings by getting that extra 20% of people to use the feature. So the whole $12 must have been produced by those switched users.

So we divide. If $12 of average effect was generated entirely by the extra 20 percentage points of adoption, then each switched user must have saved about $12 / 0.20 = $60 more per month. That ratio — the instrument's effect on the outcome divided by its effect on the treatment — is the simplest instrumental-variable estimate, called the Wald estimator.

The Wald estimator: the email's effect on savings (the reduced form, $12) divided by its effect on adoption (the first stage, 0.20) gives the savings caused by the feature itself, about $60 per month.

import pandas as pd

# z = 1 if the user was emailed (the instrument)
# used  = 1 if they adopted the feature; saved = monthly savings in dollars
first_stage  = df[df.z == 1].used.mean()  - df[df.z == 0].used.mean()    # 0.40 - 0.20 = 0.20
reduced_form = df[df.z == 1].saved.mean() - df[df.z == 0].saved.mean()   # 312  - 300  = 12
iv_estimate  = reduced_form / first_stage                               # 12 / 0.20 = 60
print(iv_estimate)   # 60  ->  savings caused by the feature, for compliers

1. Find or create an instrument Z that bumps the treatment but plausibly affects the outcome only through it (here: a randomly-sent encouragement email).
2. First stage: measure how much Z changes the treatment (here +20 percentage points of adoption). If this is tiny, the instrument is weak and the estimate becomes wildly noisy — stop here.
3. Reduced form: measure how much Z changes the outcome (here +$12 of monthly savings).
4. Divide: the IV effect is the reduced form divided by the first stage (here $12 / 0.20 = $60).

Famous real instruments work the same way. Military draft lotteries have been used to study the effect of military service on later earnings; distance to the nearest college as an instrument for years of education; rainfall as an instrument for one economic variable's effect on another. Each is prized precisely because it's hard to argue it could touch the outcome by any path other than the treatment.

Difference-in-differences: before/after, treated/control

Here's a different shape of accidental experiment. Our company rolls out free shipping in Region A starting in March, but not in neighbouring Region B. We have sales data from before and after. The temptation is to look only at Region A: weekly sales there were 100 units before March and 130 after, so free shipping must have added 30. But the world changed for everyone in March — spring arrived, a brand campaign ran — so some of that +30 would have happened regardless of free shipping.

This is exactly what Region B is for. Region B got no free shipping, yet its weekly sales rose from 90 to 110 — a +20 lift driven purely by those same seasonal forces. That +20 is our best guess of what Region A would have done on its own, without free shipping. So the part genuinely caused by free shipping is the extra rise on top of that: 30 − 20 = 10 units. We took a difference over time, and then a difference between the two differences — hence the name.

The difference-in-differences estimate: the treated group's before-to-after change minus the control group's before-to-after change.

Everything here rests on one belief we can never fully prove: that without the change, Region A would have moved in parallel with Region B. This is the parallel-trends assumption. We can build confidence in it by checking the period before March — if the two regions were already drifting apart, parallel trends is doubtful and the method is in trouble. Plotting the pre-period trends is the single most important diagnostic a difference-in-differences study can show you.

# treated = 1 for Region A, 0 for Region B;  post = 1 after the March rollout, 0 before.
# The coefficient on the treated:post INTERACTION is exactly the difference-in-differences estimate.
fit <- lm(sales ~ treated + post + treated:post, data = df)
summary(fit)$coef['treated:post', ]   # estimate = 10  (the causal lift from free shipping)

A celebrated real example: economists David Card and Alan Krueger compared fast-food employment in New Jersey, which raised its minimum wage, against neighbouring Pennsylvania, which didn't — before and after the increase. Difference-in-differences let them isolate the policy's effect, and they found employment did not fall the way simple theory predicted, a result that reshaped a long-running debate.

Regression discontinuity: just above the cutoff

Now a third kind of natural experiment, hiding inside rules with a sharp cutoff. Suppose a scholarship is awarded to every student who scores 80 or higher on an exam. Does the scholarship cause higher graduation rates, or do high scorers graduate anyway? Comparing a student who scored 95 against one who scored 60 is hopeless — they differ in a hundred ways. But here's the magic: a student who scored 79 and a student who scored 81 are essentially the same person. One point on a noisy exam is basically luck. Yet 81 gets the scholarship and 79 does not.

So right at the cutoff, who gets the treatment is as-good-as-random. The score is called the running variable, 80 is the cutoff, and the treatment switches on the instant you cross it. We fit a trend to the outcome on each side of the cutoff and measure the jump — the discontinuity — exactly at 80. Say graduation rates run at about 55% just below the cutoff and about 67% just above it: that 12-percentage-point jump is the causal effect of the scholarship, for students near the cutoff.

In words: take the outcome trend as the score X approaches the cutoff c from just above, subtract the trend approaching from just below, and the vertical gap is the effect. Here that gap is about 67% − 55% = 12 percentage points.

Real-world cutoffs are everywhere once you look. Close elections are a favourite: a candidate who wins by 0.1% versus one who loses by 0.1% is as-good-as-random, which lets researchers study the effect of holding office. The legal drinking age of 21 has been used the same way — comparing people just under 21 with people just over — to estimate the effect of alcohol access on mortality.

The assumptions each design rests on

Every one of these methods buys its causal credibility with an assumption you must argue for — not one you can simply compute your way past. Here they are in plain words, together with how to stress-test each one. Read this as the fine print on the contract.

Instrumental variables need three things. Relevance: the instrument really does move the treatment — this one is checkable, it's just the first stage, and a weak instrument (a tiny first stage) makes the estimate explode with noise. The exclusion restriction: the instrument affects the outcome only through the treatment — this is not checkable from the data at all; it is pure argument (our email must not change savings by any route except feature use). And independence: the instrument is unrelated to the confounders — a randomly-sent email satisfies this by construction, but a clever instrument like distance to college might quietly correlate with family wealth.

Difference-in-differences rests above all on parallel trends: without the treatment, the treated and control groups would have moved in parallel. This is partly checkable — plot the pre-period and look for matching trends — but never fully provable, because the counterfactual after treatment is exactly what we don't get to see. It also needs no other shock to hit only the treated group at the same moment, and a stable group composition with no spillovers from treated to control.

Regression discontinuity rests on continuity: everything else that affects the outcome must vary smoothly through the cutoff, so the only thing that jumps at 80 is the treatment. It also needs no manipulation or sorting around the cutoff — checkable with the density test above — and you must remember the estimate is local to the cutoff, full stop.

Reading a quasi-experimental study critically

You will meet these designs constantly — in news headlines, in policy reports, and in your own team's analyses. Here is a short, practical routine for reading one without being fooled. Run through it every single time, before you let yourself believe the conclusion.

1. Name the source of variation. Point to the instrument, the rollout, or the cutoff. If you can't find one, this is probably an ordinary correlational comparison wearing a fancy name.
2. Ask if it's really as-good-as-random. What else could be correlated with that variation? Did the emailed group differ to begin with? Did Region A also get a separate promotion in March? Could a student push their score over 80?
3. Look for the assumption test. A pre-trends plot for difference-in-differences; a reported first-stage strength for instrumental variables; a density/manipulation check for regression discontinuity; a balance table everywhere. A study that shows none of these is asking for blind faith.
4. Ask whose effect it is. Instrumental variables and regression discontinuity give local effects — compliers, or people near the cutoff. Don't quietly generalize them to everyone.
5. Read the interval, not just the star. A confidence interval shows the whole range of effects consistent with the data; a tiny p-value attached to a trivial effect is statistically significant yet may be practically meaningless.
6. Ask if the effect is big enough to matter. Statistical significance is not the same thing as practical importance — always translate the estimate back into real units a decision-maker cares about.

Natural experiments are among the most powerful ideas in all of data analysis. They let us make honest causal claims about the messy, un-randomizable real world — wages, schooling, health, public policy — where a clean randomized trial is out of reach. But their power is borrowed from assumptions. Used with humility and stress-tested hard, they turn the world's accidents into something close to the truth machine of a real experiment. Used carelessly, they're just correlation in a lab coat. The next guide picks up the applied toolkit — matching and propensity scores — for when no tidy natural experiment is lying around at all.