Natural Experiments and Quasi-Experimental Designs

Designs that buy identification

The gold standard is a randomized experiment — an A/B test — because randomization severs every backdoor path by construction. But experiments are often impossible, unethical, or too slow. Quasi-experimental designs exploit accidents of how treatment got assigned in the real world to approximate that randomness. The art is to find a setting where, conditional on the design's assumptions, treatment is as good as random, then lean on those assumptions explicitly rather than hoping a regression sorts confounding out.

Difference-in-differences

Difference-in-differences (DiD) is the workhorse for policy evaluation with panel or repeated cross-section data. A treated group adopts an intervention at some time; a control group never does. Comparing the treated group's before-and-after change to the control group's before-and-after change differences out any fixed, time-invariant gap between the groups and any common shock hitting both. The key assumption is parallel trends: absent treatment, the two groups' outcomes would have moved in parallel.

\hat{\tau}_{\text{DiD}} = \big(\bar{Y}^{\,\text{post}}_{\text{treat}} - \bar{Y}^{\,\text{pre}}_{\text{treat}}\big) - \big(\bar{Y}^{\,\text{post}}_{\text{ctrl}} - \bar{Y}^{\,\text{pre}}_{\text{ctrl}}\big)

The 2x2 difference-in-differences estimator: subtract the control group's before-after change from the treated group's, netting out shared time trends.

Regression discontinuity

Regression discontinuity design (RDD) exploits a hard threshold rule. A scholarship goes to students scoring above 80; a subsidy goes to firms below a size cutoff. Just above and just below the cutoff, units are essentially identical in everything except treatment — the threshold acts as a local randomizer. By comparing outcomes in a narrow window around the cutoff, RDD identifies a sharp local effect: the effect for units right at the boundary. Its credibility is high precisely because the comparison is so local; its limitation is that the estimate may not extrapolate far from the threshold.

\tau_{\text{RDD}} = \lim_{x \downarrow c}\, \mathbb{E}[\,Y \mid X = x\,] - \lim_{x \uparrow c}\, \mathbb{E}[\,Y \mid X = x\,]

Sharp RDD identifies the causal effect as the jump in the outcome exactly at the cutoff c of the running variable.

Instruments in the wild

Natural experiments often hand you an instrumental variable for free: a lottery for military draft, weather as a shifter of supply, a policy rollout staggered by geography. The instrument injects exogenous variation into treatment, and two-stage least squares recovers the effect for compliers — those whose treatment status actually responds to the instrument. This is a local average treatment effect, and being clear about whose effect you estimated matters as much as the number itself.

\tau_{\text{LATE}} = \frac{\operatorname{Cov}(Y, Z)}{\operatorname{Cov}(D, Z)} = \frac{\mathbb{E}[Y \mid Z = 1] - \mathbb{E}[Y \mid Z = 0]}{\mathbb{E}[D \mid Z = 1] - \mathbb{E}[D \mid Z = 0]}

The Wald / IV estimator: the instrument's effect on the outcome divided by its effect on treatment, identifying the local effect for compliers.

From effects to actions: uplift modeling

Estimating an effect is the means; acting well is the end. Uplift modeling predicts the individual incremental effect of an intervention — essentially the CATE repackaged for targeting — and ranks people by how much a treatment changes their outcome, not by how likely the outcome is. The classic four quadrants are illuminating: persuadables respond only if treated, sure things respond regardless, lost causes never respond, and sleeping dogs are actually harmed by the treatment. Spending only on persuadables, and never on sleeping dogs, is where the value is.

\tau(x) = \mathbb{E}\big[\,Y(1) - Y(0) \mid X = x\,\big]

Uplift modeling targets the conditional average treatment effect — the individual incremental effect of the intervention given features x.