Why we run experiments

The trap of “we shipped it and numbers went up”

Imagine you work on a shopping app. For weeks the team has argued about the checkout button: someone redesigns it from a dull grey to a bright, confident green, you ship the change to everyone on Monday, and by Friday revenue is up 8% compared with the week before. The room celebrates. The green button works. Time to redesign every button green.

Now pause and ask one quiet question: what else changed that week? Maybe it was payday. Maybe a holiday weekend was coming. Maybe marketing sent a big email on Wednesday, a competitor's site went down on Thursday, the weather turned, or a product went viral on social media. The 8% you are so proud of is the effect of the green button tangled together with every one of those other things. You cannot tell them apart from this number alone.

To truly credit the button, you would need to know what revenue would have been that same week, with the same shoppers and the same weather and the same marketing email, if you had kept the old grey button. That parallel world — same everything, one detail changed — is called the counterfactual, and here is the hard truth: you can never observe it directly. The week happened once, with the green button. The grey-button version of that exact week is gone. This gap is the beating heart of every question about cause and effect.

A “before versus after” comparison feels like evidence, but it quietly assumes that nothing except your change moved between the two periods. In the real world, something is always moving. That is why opinions about what works are cheap, and why this guide is about the one tool that gives us an honest answer: the randomized experiment.

Selection bias: why observational data lies

Most of the data a company has is observational data: data you collect by simply watching what people do, without stepping in to change anything. Server logs, sign-up records, survey answers, purchase histories — all of it is the world recording itself. It is enormous, cheap, and already sitting in your database. It is also, for causal questions, a minefield.

Here is a story you will hear in every company. An analyst notices that users who use Feature X keep coming back far more than users who don't: 80% of Feature-X users are still active after a month, versus only 30% of everyone else. The slide says: Feature X drives retention — push everyone to use it! But think about who finds and uses an advanced feature in the first place. It tends to be the already-devoted power users — the people who were going to stick around anyway. The two groups were different before Feature X ever entered the picture. The feature did not necessarily create the loyalty; loyal people sought out the feature.

This is selection bias: the groups you are comparing were sorted into those groups by something other than chance, so they differ in ways that have nothing to do with your feature. A close cousin is the confounder — a lurking third variable that influences both things you are looking at and so manufactures a correlation between them. The textbook example: across a summer, ice-cream sales and drowning deaths rise together. Ice cream does not cause drowning. Hot weather is the confounder; it independently pushes up both.

What randomization actually fixes

Here is the move that turns a guess into knowledge. Instead of letting users sort themselves into groups, you decide for them — by chance. As each user arrives, you flip a (digital) coin. Heads, they see the new green button; tails, they keep the old grey one. Nothing about who they are touches the coin. This is randomization, and it is the single most important idea in experimentation.

Why is a coin flip so powerful? Because the coin does not care about anything. Over enough users, the two groups end up balanced on every characteristic at once — age, country, device, how engaged they already were, what day they signed up, even the weather they will happen to experience. And here is the part that no amount of clever analysis of observational data can match: the groups are balanced even on the things you never measured, never logged, and never even thought of. Randomization protects you from confounders you don't know exist.

Once the two groups are alike in every way except one — which button they saw — any difference in their outcomes can finally be pinned on that one deliberate difference. The grey-button group becomes a living, breathing stand-in for the counterfactual: it shows you what the green-button group would have done if they had seen grey instead. That is why a randomized experiment is the closest thing we have to a truth machine for cause and effect.

Putting a number on the effect is then refreshingly simple. You measure the average outcome in each group and subtract. If 5.0% of green-button users bought something and 4.6% of grey-button users did, your estimated effect is a 0.4 percentage-point lift. The quantity you are estimating has a name — the average treatment effect — and in its simplest form it is just one average minus the other.

The estimated average treatment effect: the average outcome (Ȳ) among users who got the new version, minus the average outcome among users who got the old one. The little hat means “our estimate of”, because it comes from a sample, not the whole world.

Treatment, control, and a fair comparison

Two pieces of vocabulary make the rest of the field easy to read. The treatment group is the set of users who get the new thing you are testing — the green button. The control group is the set who get the existing experience — the grey button — and serve as the baseline you compare against. The treatment is the change; the control is “business as usual.”

The control group is not a formality you can skip. It is your best earthly estimate of the counterfactual — what the treatment group would have done without the change. Skip the control and run treatment-only, and you are right back in the “numbers went up” trap, unable to separate your change from the holiday, the email, and the weather. The control quietly absorbs all of those background forces, because it lived through the very same week.

For the comparison to be fair, three things must line up. The two groups must come from the same population, drawn at the same time, and differ in exactly one thing — your treatment. Change two things at once (say, the green button and a discount shown only to the treatment group) and the experiment is ruined: a lift could come from either, and you will never know which. This “change one thing” discipline is what makes an experiment interpretable.

There is one more subtle threat: people sometimes behave differently just because they know they are being watched or because they got something new and shiny. In medicine this is why the control group still gets a sugar pill — a placebo — and why neither patients nor doctors are told who got which (“blinding”), so that hope and expectation press equally on both groups. Online you rarely tell users they are in an experiment at all, but the same spirit applies: the only thing that should differ between treatment and control is the treatment itself.

The A/B test in one picture

When a tech company runs a randomized experiment on its website or app, it usually calls it an A/B test. The name is just shorthand: version A is the control (the current experience) and version B is the treatment (the new idea). As real users arrive, software randomly assigns each one to A or B — often a 50/50 split — then logs what they do. An A/B test is nothing more exotic than a randomized controlled experiment dressed in product clothing.

The skeleton of every A/B test is the same five moves. Notice that four of the five are decided before you look at a single result — experimentation is mostly planning, and the analysis at the end is the short part.

1. Pick the one metric that decides success — for a checkout button, the purchase conversion rate. Write down which direction counts as a win.
2. Choose what unit to randomize — usually each user — and split the incoming users randomly into control (A) and treatment (B).
3. Expose each group to its version and let the experiment run untouched for the planned duration.
4. Measure the chosen metric in each group and compute the difference between them.
5. Decide whether that difference is real (not just noise) and big enough to matter — using statistics, not gut feeling.

SELECT
    variant,                         -- 'A' (control) or 'B' (treatment)
    COUNT(*)                AS users,
    AVG(converted::float)   AS conversion_rate
FROM experiment_events
WHERE experiment = 'green_checkout_button'
GROUP BY variant;

That last step hides a lot. How do you know a 0.4-point gap is a genuine effect and not the random luck of which users happened to land in each group? How long must you run the test to give a real effect a fair chance of showing up? Those questions are the subject of the next guides in this track — choosing the metric and the randomization unit, then sizing the test with statistical power. For now, the picture above is the whole idea: split fairly, expose, measure, compare.

When you can't — or shouldn't — experiment

Randomized experiments are the gold standard, but they are not always possible, ethical, or wise. An honest practitioner knows their limits as well as their power. Here are the common situations where the coin flip is off the table.

1. It would be unethical. You cannot randomly assign people to smoke for thirty years to measure cancer risk. Much of medicine and economics has to learn from data the world hands it, not from trials we would refuse to run.
2. It's impossible or one-shot. You cannot A/B test a once-in-a-decade brand relaunch, a new national law, or a merger. There is only one of it, and it happens to everyone at once.
3. Interference between users (network effects). If you give a new social or messaging feature to half your users, the treatment group changes the experience of the control group — a message has two ends. The clean separation between groups breaks down.
4. It's too slow or too costly. Some effects — churn over years, lifetime value, brand trust — take far longer to appear than any product team is willing to wait, or would cost too much to test at scale.

When you genuinely cannot randomize, you are not helpless — you fall back on observational data and a careful toolkit called causal inference, which tries to approximate an experiment from data the world gave you. Sometimes the world even runs an experiment for you by accident — a policy that switches on at a date, a lottery, an arbitrary cutoff — and these natural experiments can be analyzed almost like the real thing. Those methods are powerful and beautiful, but they always rest on assumptions you must argue for and cannot fully prove. The whole Causal Inference track is devoted to doing this well.

The experimentation mindset

Experimentation is more than a statistical technique; it is a way of holding your own opinions loosely. The mindset is a loop: form a clear hypothesis, run a fair test, read the result honestly, and feed what you learned into the next idea. Teams that internalize this stop arguing about whose intuition is better and start letting reality cast the deciding vote.

Here is the most liberating fact in the whole discipline: most experiments fail. At large tech companies, only a third or fewer of tested ideas move the target metric in the right direction; many do nothing, and a painful share actively make things worse. That is not a sign the program is broken — it is the program working. Experiments are cheap ways to kill bad ideas before they ship to millions, and a clear “no” is just as valuable as a “yes.” The alternative is the HiPPO — the Highest-Paid Person's Opinion — quietly steering the product on vibes.

Two warnings to carry into the guides ahead, because they are where good experimenters earn their keep. First, a result being “statistically significant” does not mean it is important: with millions of users, even a meaningless 0.01% lift can be flagged as real, so always ask whether the effect is big enough to act on. (You will meet the related ideas of the p-value and the confidence interval soon, and you will learn to distrust the first and lean on the second.) Second, the more you peek at a running test and the more metrics you check, the more false alarms you will manufacture — discipline before the experiment is what protects you.

1. Write the hypothesis and the one success metric down before you start — a prediction you could be wrong about.
2. Randomize properly, and change exactly one thing so the result is interpretable.
3. Decide the run length and sample size up front, and don't stop early just because the numbers look good.
4. Judge both whether the effect is real and whether it is large enough to be worth the cost and complexity of shipping it.
5. Whatever happens, write down what you learned — a failed experiment that kills a bad idea is a success.

That is the case for experiments: opinions are cheap, observational data is biased, and the world rarely volunteers a clean counterfactual — so when you can, you manufacture one with a coin flip. In the next guide we get practical about design: choosing the metric that actually captures success, the unit you randomize on, and the guardrails that keep an experiment from helping one number while quietly hurting another.