Beyond simple A/B

A/B/n and factorial designs

A simple A/B test compares exactly two versions of something: the current experience (the control) and one new idea (the treatment). You split users at random, show each group one version, and compare a metric. But real life rarely offers you a tidy choice between two options. A designer shows up with four candidate checkout buttons, not one. Marketing has three subject lines they all love. The natural move is to test them all at once — and that is what an A/B/n test is: one control plus several treatments, all running side by side, traffic split among them.

Running n variants instead of two costs you in two ways, and it is worth being blunt about both. First, your traffic is now divided into more buckets, so each bucket is smaller. A smaller group means a noisier estimate, which means less ability to detect a real difference — you have quietly lowered your statistical power for every comparison. Second, and more insidiously, every extra variant is another comparison, and every comparison is another roll of the dice at producing a false positive purely by chance.

With m independent comparisons, each at significance level α, the chance of at least one false alarm. At α = 0.05 and m = 10 comparisons this is about 0.40 — a 40% chance of crying wolf even if every variant is useless.

The Bonferroni correction: to keep the overall false-alarm rate near α across m comparisons, demand each comparison clear this tighter bar instead.

There is a smarter alternative to throwing many separate variants at the wall. Suppose you want to test a new button color and a new headline. Instead of two separate experiments, run a factorial design: a small grid that crosses both changes. A 2×2 factorial has four cells — old/new color crossed with old/new headline. From the same users you learn the main effect of color, the main effect of the headline, and — the real prize — their interaction: whether the new color helps more (or less) when paired with the new headline. That is far more information than two isolated tests, for the same traffic.

Sequential testing: stopping early, correctly

Here is a temptation every team feels. You launch a test meant to run for two weeks, but on day three the dashboard already shows the treatment winning with a significant result. Why wait? Ship it. The problem is that you did not just look once — you looked on day one, day two, day three, and you stopped the moment the result crossed the line. This is peeking, and it badly inflates your false-positive rate, because every extra look is another chance for ordinary random noise to wander across the significance threshold.

Under the null hypothesis H₀ (no real difference), a test designed for a single look has false-alarm rate α. Peek repeatedly and stop at the first significant result, and the true false-alarm rate climbs far above α — often to 20–40% for a nominal 5% test.

It helps to be precise about what a p-value even is, because peeking abuses exactly this. A p-value is the probability of seeing data at least this extreme if the null hypothesis — no real effect — were true. It is emphatically not the probability that your variant is better, nor the probability the null is true. The whole machinery of a classical test assumes you fix the sample size in advance and look once. Peek many times and you have quietly changed the experiment, so the p-value no longer means what the table says it means.

But teams genuinely need to monitor a live test — to kill a harmful variant fast, or to ship a clear win sooner. The honest way to do this is sequential testing: a family of methods built from the start for continuous monitoring. Two ideas you will meet. Alpha-spending (group sequential designs) treats your 5% false-alarm budget like money: you spend a little of it at each scheduled look, so the thresholds early on are stricter and the total stays controlled. Always-valid inference produces p-values and confidence sequences that remain trustworthy no matter when you choose to stop — you may look as often as you like.

CUPED: using pre-experiment data to cut the noise

Most of the variation in a typical metric has nothing to do with your experiment. Imagine you are testing a change to a shopping app and your metric is revenue per user. One user spent $500 last month and one spent $2, and they will keep behaving roughly like that no matter which button you show them. That huge pre-existing spread between users is noise that swamps the small signal you are hunting for — the few extra dollars your new button might earn.

CUPED — short for Controlled-experiment Using Pre-Experiment Data — is a clever trick to remove that predictable part. For each user you also look at their value before the experiment started (say, last month's spend). Because a heavy spender before tends to be a heavy spender during, you can subtract out the part of their behavior you could have predicted from the past, and analyze only what is left. The treatment effect is untouched, but the noise shrinks.

The adjusted metric. Y is the user's value during the experiment, X is their value before it, X̄ is the pre-period average, and θ is a number chosen to remove as much noise as possible. Subtracting a quantity that does not depend on the treatment leaves the effect unbiased.

How much noise you remove. ρ is the correlation between the before and during values. If they correlate at 0.7, you cut the variance by about half (1 − 0.49 ≈ 0.51) — for free, just by using data you already had.

Why does cutting variance matter so much? Because the number of users you need is driven by it. The required sample size grows with the variance of your metric and shrinks with the square of the effect you want to catch. Halve the variance and you roughly halve the users you need — or, equivalently, run the same test in half the time. CUPED is one of the most beloved tricks in industry precisely because it hands you extra power without collecting a single extra user.

The sample-size intuition: users needed (n) rise with the metric's variance σ² and fall with the square of the smallest effect δ you care about. CUPED attacks the σ² in the numerator directly.

Multi-armed bandits: earn while you learn

A classic A/B test is pure learning: you commit to a fixed split — say 50/50 — and you hold it for the entire run, even after one side is obviously losing. That is the right thing when your goal is a trustworthy measurement. But sometimes your goal is different: you mostly want to earn the most reward while you figure out which option is best. A one-day flash-sale headline, or which of five promotional emails to send, is about results today, not a clean estimate for the archive.

This is the explore–exploit tradeoff, and the tool for it is the multi-armed bandit. The name comes from slot machines — a single slot machine is a one-armed bandit, so a row of them with different unknown payouts is a multi-armed bandit. You must keep exploring arms to learn which pays best, while exploiting the arm that currently looks best so you do not throw away reward. A bandit algorithm does this dynamically: as evidence builds that one variant is winning, it automatically sends more traffic that way and starves the losers — instead of stubbornly feeding half your users to a known loser for two weeks.

Two common recipes give the intuition. Epsilon-greedy is the simplest: most of the time send users to the arm that looks best so far, but a small fraction of the time (epsilon) pick an arm at random, just to keep learning. Thompson sampling is more elegant: keep a belief about each arm's true rate, draw one random sample from each belief, and route the user to whichever sample came out highest. Arms you are unsure about get explored naturally; arms that are clearly worse quietly fade.

Holdouts and measuring long-term effects

An experiment usually runs for a week or two, but the effect you actually care about is the one that lasts for months. The short window can mislead you in predictable ways. A flashy new feature draws a curiosity bump — the novelty effect — that fades once it stops being new. The opposite, a primacy effect, happens when users are confused at first and need time to learn the change, so a genuinely good feature looks bad for a while. Either way, what you measure in week one is not what you get in month six.

The fix is a holdout group: a slice of users — often just 1% — that you deliberately keep on the old experience for a long time, weeks or months, even as you ship the new version to everyone else. Later you compare the held-out users with the rest, and that gap is your honest, durable estimate of the cumulative effect of everything you launched. A variant of this is the long-term holdout or reverse experiment, where you temporarily turn an already-shipped feature back off for a small group to re-measure its real ongoing value.

A concrete picture: a recommendation change shows a thrilling +5% engagement in week one. You ship it, but you keep a 1% holdout. Three months later the holdout comparison shows the durable lift is only +1% — still real, still worth having, but a fifth of what the launch report bragged about. Without the holdout, your roadmap would be built on a number that quietly evaporated.

Building an experimentation culture

Every technique in this guide is just a tool, and tools do not save a team that does not trust its own experiments. The companies that learn the fastest are not the ones with the cleverest statistics — they are the ones who made running a trustworthy experiment the boring default. That is a cultural achievement more than a technical one, and it is built from a handful of habits.

1. Build one central platform. A single, shared system for spinning up tests handles randomization, logging, and analysis the same way every time — so results are comparable and nobody reinvents the wiring (or its bugs).
2. Be trustworthy by default. Bake in the safety checks: automatic sample-ratio mismatch alarms, guardrail metrics that block obvious harm, a pre-set significance level, and statistics that handle peeking correctly.
3. Pre-register the plan. Before launch, write down the hypothesis, the primary overall evaluation criterion, and the minimum detectable effect. Deciding what counts as success in advance is the cheapest defense against fooling yourself afterward.
4. Keep institutional memory. A searchable log of every past experiment and its outcome stops the team from re-running the same idea and lets new wins build on old knowledge.
5. Teach and review. Peer-review experiment designs the way you review code, and teach newcomers the traps — peeking, multiple comparisons, novelty effects — before they fall into them.
6. Keep honest score. Report how often ideas fail, not just the wins. A culture that celebrates a clean negative result as a saved mistake is a culture that learns.

That last habit deserves emphasis, because it is the most counter-cultural. Teams at companies like Microsoft and Google have reported that only a minority of well-designed experiments — often something like a third or fewer — actually move the target metric in the intended direction. That is not a failure of the program; it is the whole point. Most plausible-sounding ideas simply do not work, and the value of experimentation is that it catches those before they ship as confident nonsense.

Where to go deeper

You now have the advanced experimentation toolkit. When two buckets are not enough, A/B/n and factorial designs test many ideas at once — if you correct for the extra comparisons. When you want to monitor a live test honestly, sequential testing lets you stop early without inflating false positives. When the metric is noisy, CUPED cuts the variance for free using data you already had. When earning beats measuring, a multi-armed bandit shifts traffic toward the winner as it learns. And when the short term lies, a holdout group tells you the truth months later.

These ideas also connect outward to the rest of data science. When you genuinely cannot randomize, the causal-inference track shows how to still make a defensible causal claim — and reminds you, again, that correlation is not causation. CUPED's covariate adjustment is really just regression in disguise, so the regression track deepens it. And every experiment you run produces data that has to be stored, cleaned, and served, which is what the modern-data-stack track is about.

For a single book that pulls the whole experimentation discipline together, the standard reference is Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang, and Ya Xu — practitioners who built these systems at Microsoft, Google, and LinkedIn. Pair it with the experimentation engineering blogs of large tech companies, where the traps in this guide are recounted as real, expensive war stories. Read them, then go run a clean experiment of your own.