Causal inference in practice

Matching: building a fair comparison group

By now you know the central problem of this whole track: a causal claim is a claim about a counterfactual — what would have happened to the very same people if we had treated them differently. The gold standard for getting that counterfactual is a randomized controlled trial, because randomization makes the treated and untreated groups interchangeable on average. But very often you cannot randomize: it would be unethical, too expensive, or the thing already happened. This guide is about the practical toolkit for those situations — and, just as importantly, about being honest about what that toolkit can and cannot promise.

Start with the most intuitive method: matching. Imagine you want to know whether a free job-training program raises monthly wages. People who enrol are not a random slice of the population — they tend to be younger, less experienced, and currently earning less, which is often why they signed up. If you just compare the two groups as they are, you are comparing apples to oranges. Suppose the raw numbers come out like this: trainees earn NT$24,000 a month afterwards, while non-trainees earn NT$26,000. Taken at face value, training appears to LOWER wages — which is almost certainly an artefact of who chose to enrol.

Matching fixes this by refusing to compare the groups wholesale. Instead, for each trainee you go and find one (or several) non-trainees who look just like them on the background characteristics you can measure — same age band, same years of education, same prior wage, same region. You then compare each trainee only to their look-alikes. After matching on those variables, the comparison might flip: matched trainees now earn NT$300 a month MORE than their matched controls. The naive comparison and the matched comparison disagree because the groups were never comparable to begin with — exactly the selection bias story from earlier in this track.

A few practical terms. Exact matching means the look-alike must agree exactly on every variable; it is clean but quickly impossible once you have many variables. Nearest-neighbour matching picks the closest available control by some distance. A caliper is a rule that says “if no control is within this distance, leave the treated unit unmatched rather than force a bad pair.” And common support (or overlap) is the requirement that, for the kinds of people you are studying, both treated and untreated examples actually exist — you cannot match a 22-year-old trainee to a non-trainee if literally no young non-trainees exist in your data.

Once matched, the estimate is almost embarrassingly simple: take the average outcome difference within the matched pairs. When you match each treated unit to a comparable control, you are usually estimating the average treatment effect on the treated (ATT) — the effect for the kind of people who actually got the treatment, which is not always the same as the effect for everyone.

The matched estimate of the effect on the treated: for each of the N_t treated units i, subtract the outcome of its matched control m(i), then average. Compare this with the average treatment effect you met earlier.

Propensity scores

Matching on a handful of variables is fine. But real problems have many confounders — age, education, prior wage, region, industry, household size, prior unemployment spells, and on and on. With twenty variables, the chance of finding a control who matches a treated unit exactly on all of them collapses to nearly zero. This is the curse of dimensionality: as you add dimensions, the space empties out and nothing is close to anything.

The propensity score is a beautiful shortcut around this. Instead of matching on all twenty variables at once, you collapse them into a single number: each person’s probability of receiving the treatment, given their measured characteristics. Call it e(x). A person with e(x) = 0.30 is someone who — based on their background — had a 30% chance of enrolling. The deep result (due to Rosenbaum and Rubin) is that if two people have the same propensity score, then within that score the treated and untreated are balanced on all the variables that went into it. So you only ever need to match, stratify, or weight on this one number.

The propensity score is the probability of being treated (T = 1) given the measured characteristics X = x. In practice you estimate it — most simply with a logistic regression of the treatment on the covariates.

import statsmodels.formula.api as smf

# 1) Estimate the propensity score: predict TREATMENT from covariates
ps_model = smf.logit(
    "enrolled ~ age + education + prior_wage + region",
    data=df,
).fit()
df["pscore"] = ps_model.predict(df)

# 2) Inverse-probability weights
df["w"] = df["enrolled"] / df["pscore"] + \
          (1 - df["enrolled"]) / (1 - df["pscore"])

# 3) Compare weighted average wages (then ALWAYS check balance + overlap)

There are three common ways to use the score. Propensity-score matching pairs each treated unit with an untreated one of nearly identical score — this is the propensity-score matching you will hear named most often. Stratification slices everyone into, say, five buckets by score and compares treated vs untreated within each bucket. Inverse-probability weighting (IPW) keeps everyone but re-weights them: rare-but-treated and rare-but-untreated people get upweighted, so the weighted treated and control groups end up looking like the same pseudo-population.

Inverse-probability weighting in one line: each treated outcome is divided by its probability of being treated, each control outcome by its probability of being a control. Dividing by a small probability upweights the rare cases.

Two checks separate a credible propensity analysis from a fragile one. First, balance: after matching or weighting, the covariate distributions in the two groups should actually look alike — analysts report standardized mean differences and want them small. The whole point was balance, so you must verify you achieved it. Second, overlap (positivity): every kind of person must have a real chance of being in either group. If some people have a propensity score of essentially 1 — they were always going to be treated — there is no comparable control for them, and dividing by a near-zero probability in IPW makes the estimate explode.

And here is the honest punchline of the whole section: propensity scores solve the dimensionality problem, not the knowledge problem. They balance the variables you measured and modelled — exactly the same variables matching would have used. A propensity score built from age and education tells you nothing about motivation if motivation was never in the data. The method is more elegant; the fundamental assumption is identical.

Sensitivity to unmeasured confounding

Every method in the last two sections rests on one assumption that you cannot check: that there is no important unmeasured confounder. Statisticians call this conditional ignorability or selection on observables, but the plain meaning is simpler — “I have measured everything that affects both who got the treatment and what happened to them.” You can never prove this from the data, because the missing variable is, by definition, missing. So the mature move is not to assert the assumption and move on. It is to ask: how strong would a hidden confounder have to be to overturn my conclusion?

This is sensitivity analysis, and it turns an unanswerable yes/no question into a useful quantitative one. The classic example is smoking and lung cancer in the 1950s. There was no randomized trial — you cannot assign people to smoke for thirty years. Critics said genetics might be a hidden confounder causing both the urge to smoke and the cancer. The statistician Jerome Cornfield answered with a sensitivity argument: for a hidden factor to explain away the observed association — where smokers had on the order of nine times the cancer rate — that factor would itself have to be more than nine times as common in smokers as in non-smokers. No plausible candidate came close. The causal claim survived not because the assumption was proven, but because it would take an implausibly strong hidden confounder to break it.

A modern, ready-made version of this idea is the E-value. The E-value of a result is the minimum strength of association — on the risk-ratio scale — that an unmeasured confounder would need to have with BOTH the treatment and the outcome, beyond the variables you already adjusted for, to fully explain away your estimated effect. A large E-value means your finding is robust: only a very strong hidden confounder could erase it. A small E-value means your finding is fragile: even a mild unmeasured factor could account for it. You report the E-value right next to your estimate, so a reader can judge for themselves.

Pre-analysis plans and discipline

Observational causal work is dangerous for a reason that has nothing to do with confounders: there are too many defensible choices. Which covariates to adjust for? Which matching method, which caliper, which outcome window, which subgroup? Each is a fork in the road, and most analysts will — entirely sincerely — keep choosing forks until the result looks clean. Andrew Gelman calls this the garden of forking paths. The result is the same disease you met in the A/B testing track under the names peeking and multiple comparisons: if you try enough specifications, one of them will be “significant” by luck alone.

The cure is borrowed from clinical trials: a pre-analysis plan (PAP). Before you look at any outcomes, you write down — and ideally time-stamp or publicly register — exactly what you will do. This converts your analysis from “whatever survived the search” into “a single test I committed to in advance.” It is the same logic as fixing your hypothesis test before collecting data, applied to the messier world of observational design.

1. State the causal question and the exact hypothesis, before seeing outcomes.
2. Define the population, the treatment, and the single primary outcome (list secondary ones separately).
3. Draw the DAG and name exactly which variables you will adjust for — and which you will deliberately NOT adjust for.
4. Fix the method in advance: matching scheme, propensity model, weighting, and the standard-error approach.
5. Pre-specify the sensitivity analysis and what robustness you would require to believe the result.
6. Label everything else as exploratory — still valuable, but hypothesis-generating, not confirmatory.

Combining experiments and causal models

In real practice these tools are not rivals; you mix them. The first question is always the same: can you run an experiment? If you can randomize even part of the problem — a randomized trial or an online A/B test — do it, because randomization is the one move that handles unmeasured confounders for free. When a full experiment is impossible, you reach for the natural experiment designs from the previous guide — an instrumental variable, difference-in-differences, or regression discontinuity — which exploit a quasi-random nudge that the world handed you. Only when none of those is available do you fall back on adjustment by matching or propensity scores, paired with a sensitivity analysis.

The most powerful move of all is triangulation: attack the same question with several methods that fail in different ways, and see whether they agree. An A/B test, a difference-in-differences on historical data, and a propensity-matched analysis each lean on different assumptions; if all three point to roughly the same effect, your confidence should rise far more than any one of them could justify alone. When they disagree, that disagreement is itself information — it tells you which assumption is doing the heavy lifting.

Experiments and causal models also help each other directly. A small, clean experiment can be used to calibrate or validate a cheaper observational pipeline you intend to run at scale: if your observational method reproduces the experiment’s answer where you have both, you can trust it more where you only have observational data. And even inside a randomized experiment, the DAG earns its keep — it tells you to check that randomization actually held, to handle people who dropped out or did not comply, and to avoid “fixing up” the analysis by adjusting for a variable measured AFTER the treatment, which can reintroduce bias.

A decision guide: which method when

Let us turn the whole track into one practical ladder. Read it from the top; stop at the first rung you can actually stand on. The higher the rung, the fewer untestable assumptions you are asking your reader to swallow.

1. Can you randomize? If yes, run a randomized experiment or A/B test. This is the top rung; nothing below it is as trustworthy.
2. Is there quasi-random variation in the world? Use an instrumental variable, difference-in-differences, or regression discontinuity — and state the assumption each one needs.
3. Do you genuinely believe you measured the key confounders? Then adjust — regression, matching, propensity scores, or IPW — and report a sensitivity analysis alongside.
4. If even that is shaky, be explicit that the result is descriptive or exploratory — an association, a lead, not a causal claim.
5. At every rung: draw the DAG FIRST, so you know what to adjust for — and, just as crucially, what to leave alone.

That last point deserves a warning, because the most common rookie mistake in applied causal work is “control for everything you have.” More covariates is not automatically safer. Adjusting for a mediator — a variable on the very path from treatment to outcome — erases part of the effect you were trying to measure. Worse, adjusting for a collider — a variable that the treatment and the outcome both cause — actively manufactures a spurious association out of nothing, the cousin of selection bias. The DAG, not the size of your spreadsheet, tells you which variables belong in the model.

One more reminder that runs through every rung: statistical significance is not the same as a causal effect, and neither is the same as practical importance. A tight confidence interval around a precisely-estimated number is worthless if that number is the answer to the wrong (non-causal) question. Precision and causality are different virtues; you need both.

The humility causal inference demands

If there is one attitude this track is trying to install, it is earned humility. A causal claim from observational data is never unconditional. It always comes attached to assumptions you could not test: that you measured the right confounders, that there is overlap, that no collider snuck into the model, that the design’s special assumption (parallel trends, a valid instrument, a sharp cutoff) actually holds. Good practice does not hide those assumptions — it lists them, defends them, and shows, through sensitivity analysis, how much hidden bias the conclusion could survive.

This is also why the phrase correlation is not causation is the beginning of the conversation, not the end. The grown-up version is: correlation becomes credible evidence for causation under stated conditions and a defensible design. The work of this whole track has been to make those conditions explicit, to give you designs that meet them, and to give you the honesty to admit when they are not met.

When you cannot run the perfect experiment — and usually you cannot — the strongest case for causation is convergent. Epidemiologists summarize the instinct in a handful of viewpoints, sometimes called the Bradford Hill considerations: a large effect is harder to explain away than a small one; a dose-response pattern (more exposure, more outcome) is reassuring; a plausible mechanism helps; consistency across different populations and methods matters most of all. No single one is a proof, and none is a checkbox to tick — but together they describe what a believable causal story feels like.

That is the end of the causal track — and, fittingly, it ends not with a formula but with a posture. The hardest question in data is whether X truly causes Y, and the honest answer is almost always “here is our best estimate, here are the assumptions it rests on, and here is how badly it would have to be wrong for our conclusion to change.” Carry that sentence into every analysis you do, and you will have learned the most valuable thing this subject has to teach.