Sample size & statistical power

The four dials: effect, noise, significance, power

Suppose you have just redesigned the checkout button on your online store. You think the new design will nudge a few more visitors into buying — but you are not certain, and you definitely do not want to ship a change that quietly makes things worse. The professional way to find out is an A/B test: show the old button to half of your visitors (the control group) and the new button to the other half (the treatment group), wait, then compare how each group behaved. The very first practical question — the one this whole guide answers — is deceptively simple: how many visitors do you need in each group, and therefore how long must you run the test?

It is tempting to answer “run it until the numbers look good.” That is the single most expensive mistake in experimentation, and much of this guide exists to replace it with something honest. The length of an experiment is not a matter of taste or patience — it is determined, almost mechanically, by four quantities you decide in advance. Working them out is called a power analysis, and the number it produces is your sample-size calculation. Get into the habit of doing it before you write a single line of experiment code.

Here are the four dials. Read them slowly — almost every confusion about experiments later on comes from blurring two of these together.

The effect you care about. Not the effect you hope for — the smallest improvement that would actually change your decision to ship. This is the minimum detectable effect (MDE). Smaller targets cost more data.
The noise. How much your metric naturally bounces around from person to person, even with no change at all. We measure it with the variance, or its square root, the standard deviation. Noisier metrics need more data.
The false-alarm rate. Your tolerance for declaring a winner that is not real. This is the significance level α, by long convention set to 5%. Crying wolf is called a type I error.
The power. The chance of catching a real effect of your target size if it truly exists. Convention is 80%. The flip side — missing a real effect — is a type II error, and the power is exactly one minus that risk. This is the dial people forget.

The whole calculation in one picture. The left bell curve is what your metric looks like if the new button does nothing (the null). The right curve is what it looks like if the true effect equals your MDE. The vertical line is the decision threshold. The small red tail of the left curve past it is α (false alarms); the shaded part of the right curve past it is the power; the rest of the right curve, stranded on the wrong side, is β — the chance of missing a real effect.

Minimum detectable effect: what's worth catching

Every sample-size calculation starts with a number that statistics cannot give you — you have to supply it. It is the smallest effect that would actually be worth acting on. Statisticians call it the minimum detectable effect, or MDE, and the name is a little misleading: it is not the smallest effect your test can possibly notice, but the smallest effect you have decided is big enough to care about and to power your test to detect reliably.

Make it concrete. Say that today, out of every 100 visitors who reach your checkout page, about 5 complete the purchase — a baseline conversion rate of 5%. You sit down with your product team and ask: what is the smallest lift that would make shipping this new button worthwhile, once you account for the engineering cost and the small risk of bugs? Suppose you agree the answer is half a percentage point — moving from 5% to 5.5%. That 0.5 percentage points is your MDE.

Notice what just happened: you set the MDE by business reasoning, not by statistics. The honest way to choose it is to ask what the effect is worth. If even a tiny 0.1-point gain would earn millions and the change is free to ship, you may want to detect very small effects — and pay for it with a huge sample. If only a sizeable jump would justify the maintenance burden, you can set a larger MDE and finish far sooner. The MDE is a decision about value, dressed up as a number.

Where the MDE lives. Visitors flow down a funnel — arrive, view cart, reach checkout, purchase — and at each step some drop off. Your baseline 5% is the width of the final arrow. The whole experiment is a bet that the new button widens that last step, even slightly. The MDE is how much widening you have decided is worth chasing.

Why smaller effects need bigger samples

Here is the intuition before any algebra. Detecting an effect is like hearing someone speak. A loud shout is easy to make out even in a crowded café; a faint whisper gets lost in the same noise. To be sure you heard a whisper, you either need a quieter room (less noise) or you need to listen for much longer (more data). A small MDE is a whisper. The room — your metric's natural variability — is as noisy as it is. So the only dial left to turn is the amount of data.

Why does more data help? Because averages settle down as you collect more of them. The spread of a sample average — how much it would jump around if you reran the experiment — is its standard error, and it shrinks in a very specific way: in proportion to one over the square root of the sample size. This single fact, which falls out of the central limit theorem, is the engine of the whole calculation.

\text{SE} \;=\; \frac{\sigma}{\sqrt{n}}

The standard error of an average equals the metric's standard deviation σ divided by the square root of the sample size n. Quadruple n and the noise in your estimate halves — not quarters. Progress is real but slow, which is exactly why precision is expensive.

Now combine the two ideas. To call an effect real, your signal (the MDE, call it Δ) has to stand out against the standard error. Since the standard error shrinks like one over root-n, the sample size you need grows like the variance divided by the square of the effect you are chasing.

n \;\propto\; \frac{\sigma^{2}}{\Delta^{2}}

The sample size per group grows with the variance σ² and, crucially, with one over the square of the MDE Δ. That square is the punchline of this whole section.

A back-of-envelope sample-size formula

The proportionality “n grows like σ²/Δ²” becomes an actual number once you pin down the false-alarm rate and the power. For the standard choices — a 5% significance level and 80% power — the proportionality constant works out to almost exactly 16. This gives a famous rule of thumb, sometimes called the rule of 16, that you can do on the back of an envelope.

n \;\approx\; \frac{16\,\sigma^{2}}{\Delta^{2}} \qquad \text{(per group)}

For a metric with standard deviation σ and a target effect Δ, you need roughly 16 σ²/Δ² observations in each group — both the control and the treatment — at the usual 5% / 80% settings.

When your metric is a yes/no outcome — bought or did not buy, clicked or did not click — there is an even friendlier version, because the variability of a proportion is fixed by the proportion itself: it is p times (1 − p). Plugging that in for σ² gives the form most A/B calculators use under the hood.

n \;\approx\; \frac{16\,p\,(1-p)}{\Delta^{2}} \qquad \text{(per group)}

For a conversion rate p with an absolute MDE of Δ, this is the per-group sample size. Here p is the baseline rate written as a fraction (5% is 0.05) and Δ is the absolute lift you want to detect (0.5 points is 0.005).

Run our checkout example through it. The baseline is p = 0.05, so p(1 − p) = 0.05 × 0.95 = 0.0475. The MDE is half a percentage point, Δ = 0.005, so Δ² = 0.000025. The arithmetic is clean:

n \;\approx\; \frac{16 \times 0.0475}{0.000025} \;=\; \frac{0.76}{0.000025} \;=\; 30{,}400 \quad \text{per group}

You need about 30,400 visitors in the control group and another 30,400 in the treatment group — roughly 60,800 in total — to have an 80% chance of detecting a true 0.5-point lift at the 5% level.

Now feel the inverse-square law bite. If you relax your MDE to a full percentage point (Δ = 0.01), the denominator becomes 0.0001 and the per-group sample drops to 16 × 0.0475 / 0.0001 = 7,600 — exactly one quarter. Caring about half as small an effect cost you four times the data. That single trade-off, effect versus sample, is the heart of experiment planning.

p, mde = 0.05, 0.005            # baseline rate, absolute MDE
n = 16 * p * (1 - p) / mde**2    # rule of 16, per group
print(round(n))                 # 30400 per group  ->  ~60800 total

# In real work, reach for a library that uses the exact z-values
# and lets you change alpha/power, e.g. statsmodels in Python:
# from statsmodels.stats.power import NormalIndPower

The rule of 16 in three lines. It is close enough for planning conversations; switch to a proper power library (statsmodels in Python, pwr in R) the moment you want to vary α or power, or handle non-50/50 splits.

Power: the chance of catching a real effect

We have leaned on the word power without fully unpacking it. Here is the clean definition: statistical power is the probability that your test will declare a result significant when there really is an effect of the size you targeted. Power of 80% means that if the new button truly lifts conversion by your MDE, you have an 80% chance of catching it — and, uncomfortably, a 20% chance of running the whole experiment and concluding “no effect” anyway.

That 20% miss has a name. The two ways a test can be wrong sit on opposite shoulders. A type I error is a false alarm: there is no real effect, but noise pushes you to declare a winner. Its rate is α, the significance level. A type II error is a miss: there is a real effect, but you fail to detect it. Its rate is called β, and power is simply one minus β.

\text{power} \;=\; 1 - \beta, \qquad \alpha = \Pr(\text{false alarm}), \qquad \beta = \Pr(\text{miss})

The two error rates and their relationship to power. By convention we cap the false-alarm rate α at 5% and aim for power of 80% (so β = 20%). Note the asymmetry: we treat a false alarm as four times more serious than a miss — a choice, not a law.

Look back at the two-curve figure from the first section, because it draws all of this at once. The decision threshold is fixed by α — slide it left and you catch more real effects (more power) but raise the false-alarm rate too. The two curves move apart when the true effect is larger, or when more data makes each curve narrower; either way the overlap shrinks and power rises. Sample size is, quite literally, the dial that squeezes those bell curves thin enough to pull apart.

Running the calculator on a real example

Let's walk the checkout test end to end, the way you would on a Monday morning, and turn the sample size into a calendar date — because “30,400 per group” means nothing to your manager until it becomes “run it for two weeks.”

Pin the baseline. Pull last month's data: 5.0% of checkout visitors purchase. That is your p = 0.05, and it also fixes the noise, since for a proportion the variance is just p(1 − p).
Choose the MDE from value. The team agrees a 0.5-point absolute lift (to 5.5%) is the smallest win worth shipping. So Δ = 0.005.
Fix α and power. Stick with the conventions: α = 5%, power = 80%. Write them down so nobody renegotiates them after seeing the result.
Compute the per-group size. The rule of 16 (or the calculator below) gives about 30,400 per group, roughly 60,800 visitors total.
Convert to time. If 10,000 visitors a day reach checkout and you split them 50/50, each group gains 5,000 per day. 30,400 ÷ 5,000 ≈ 6.1 days — but round up to a full week or two to absorb the weekday/weekend rhythm.
Commit before you start. Decide the run length now and do not stop early just because the numbers look good. (The next guide explains exactly why peeking betrays you.)

Type your own numbers in. Set the baseline rate, the MDE, the significance level and the power, and the calculator returns the sample size you need per group — and, if you give it your daily traffic, how many days that is. Try halving the MDE and watch the required sample roughly quadruple, exactly as the inverse-square law predicts.

Common sizing mistakes

You now know the machinery. The last job is to recognise the ways it gets quietly broken in practice — because almost every one of these mistakes makes a test look more conclusive than it is.

Not fixing the sample size in advance. If you decide the duration after watching the data, you have abandoned the calculation entirely. Compute n first, then start.
Stopping the moment it looks significant — peeking. Checking repeatedly and stopping at the first significant reading can push your true false-alarm rate from 5% to 20% or more. Pick the end date in advance and honour it (or use a proper sequential method).
Confusing absolute and relative MDE. Feeding a 10% relative lift into a formula that expects a 0.005 absolute lift will mis-size the test by a factor of a hundred. Decide which you mean and keep it consistent.
Forgetting that n is per group. The formula gives the size of each arm; the total visitors you must collect is at least double that.
Using the effect you hope for as the MDE. Optimistic MDEs produce reassuringly small samples and badly underpowered tests. Size for the smallest effect worth detecting, not the one in your slides.
Watching twenty metrics at once. Test enough outcomes and one will cross 5% by pure chance — the multiple-comparisons trap. Name one primary metric up front; treat the rest as exploratory or as guardrail metrics.

Plan the four dials honestly, solve for the sample size before you start, and let the experiment run its full course — and you have done the hardest, most respected part of A/B testing. With the test correctly sized, the next guide turns to the equally treacherous job of reading the result without fooling yourself.