Reading results — and the traps

Read the interval, not just the p-value

You designed the test carefully, ran it long enough, and now the dashboard shows two numbers side by side. The temptation is to glance at one cell — “p < 0.05, significant, ship it!” — and move on. This guide is about everything that can go wrong between “the result is in” and “we made the right call.” We start with the single most useful habit: read the confidence interval on the difference, not just the p-value.

Here is a concrete case we will carry through the whole guide. You are testing a new checkout button. The control group sees the old button; the treatment group sees the new one. Each arm got 30,000 visitors. Control converted at 5.0%; treatment converted at 5.4%. The headline difference is +0.4 percentage points — a relative lift of about +8%. The p-value comes out around 0.024, which means: “if the new button truly made no difference, a gap this big or bigger would be fairly unlikely — about a 2.4% chance.” Fine. But that one number throws away the thing you actually need to decide: how big is the effect, and how sure are we?

The confidence interval keeps that information. For this result the 95% interval on the difference is roughly +0.05 to +0.75 percentage points — in relative terms, about +1% to +15%. That interval is the honest summary: the true lift is plausibly anywhere from “barely worth the trouble” to “genuinely great.” A bare “significant” hides the fact that the low end sits almost on top of zero.

Where does the interval come from? First write the quantity we care about — the difference between the two observed conversion rates (the little hat means “estimated from the sample”):

The effect we are estimating: treatment rate minus control rate.

Every estimate carries noise. The standard error of the difference combines the wobble from each arm — for a proportion, each arm’s noise depends on the rate and the sample size:

The combined standard error of a difference between two proportions.

Plug in the numbers and the standard error of the difference is about 0.18 percentage points. The 95% interval reaches roughly 1.96 standard errors on each side, so the margin is about 0.35 points: 0.40 ± 0.35 gives +0.05 to +0.75 points. Because the lower edge sits just above zero, the p-value squeaks under 0.05. The interval and the p-value tell the same story — but the interval tells you more.

An approximate 95% confidence interval on the effect.

Statistical vs practical significance

Statistical significance answers one question: “is this gap probably more than noise?” Practical significance answers a completely different one: “is the effect big enough to be worth acting on?” These two come apart constantly, and confusing them is one of the most common mistakes in reading a result.

With enough users, almost any difference becomes statistically significant. Imagine 5 million visitors per arm and a +0.01 percentage-point difference — utterly trivial. The standard error is so tiny that even this microscopic gap produces a p-value far below 0.05. Significant? Yes. Worth a quarter of engineering rework, a more complex codebase, and the risk of new bugs? Almost certainly not.

The flip side bites too: a genuinely large, valuable effect can fail to reach significance if the sample is small or the metric is noisy. That is exactly why you ran a power analysis and fixed a minimum detectable effect before launching — so the test had a real chance of catching an effect worth catching. A “not significant” result on an underpowered test means “we couldn’t tell,” not “there is no effect.”

So how do you judge practical significance? Decide, before the test, the smallest lift that would actually change your decision — that is your MDE. Then compare the whole confidence interval to that line. The relative lift you are comparing is just the difference divided by the baseline:

The effect expressed as a percentage of the baseline — usually easier to reason about than raw points.

1. If the ENTIRE interval sits above your “worth it” threshold — a clear, actionable win.
2. If the interval straddles the threshold — the test genuinely can’t tell you; you need more data or a bigger effect.
3. If the entire interval is below the threshold — even when “significant,” the effect is too small to act on.

Peeking and why it inflates false positives

Here is a habit that feels responsible and quietly ruins your test: opening the dashboard every morning and deciding to stop the moment the p-value dips below 0.05. This is called peeking, and it inflates your false-positive rate far above the 5% you think you are running at.

Why does looking hurt? The 5% guarantee of a standard test assumes you look once, at a sample size you fixed in advance. Every extra look is another roll of the dice — another independent chance for random noise to wander across the 0.05 line. If your rule is “stop at the first crossing,” you are not measuring the effect; you are cherry-picking the moment noise happened to look biggest.

An honest analogy: flip a fair coin in bursts and watch the running tally. At some point, purely by luck, heads will be temporarily ahead by what looks like “a lot.” If you are allowed to call “the coin is biased!” at any moment you choose, you will eventually call it — on a perfectly fair coin. Peeking gives noise that same freedom.

The numbers are brutal. Look once, as planned, and your false-positive rate is the intended 5%. Peek five times along the way and stop at any crossing, and it climbs to roughly 14–19%. Monitor continuously and stop the instant you ever cross 0.05, and under a true null the false-positive rate creeps toward 100% — given enough time, a fair coin will look “significant” at some point.

Sample-ratio mismatch: your test is broken

You split traffic 50/50, so when the test ends you expect roughly equal counts in each arm. If instead you see a meaningful imbalance — say 50,500 versus 49,500 out of 100,000 — something in the plumbing is broken. This is sample-ratio mismatch (SRM), and it is the single most important sanity check before you interpret anything.

Why is a small imbalance a red flag rather than a curiosity? Randomization is the entire reason the two groups are comparable in the first place — it is what lets you treat the control as a fair stand-in for “what would have happened anyway.” If the ratio is off, your assignment or your logging is broken, which usually means the two groups now differ in some hidden, non-random way. Once that is true, every other number — your effect, your p-value, your interval — is built on sand.

You detect SRM with a chi-square test that compares the observed counts to the expected split. The statistic adds up, for each arm, the squared gap between observed and expected, scaled by what was expected:

The chi-square goodness-of-fit statistic; O is observed count, E is expected count.

from scipy.stats import chisquare

observed = [50500, 49500]      # users actually bucketed into A and B
expected = [50000, 50000]      # what a clean 50/50 split should give

chi2, p = chisquare(observed, expected)
print(chi2, p)                 # 10.0  0.00157  -> SRM! stop and debug

Notice how sensitive this is. A gap that looks like a rounding error to the naked eye — 50,500 vs 49,500 — produces a p-value around 0.0016 at this scale. Because trivial deviations show up at large n, teams use a strict bar for SRM (often p below 0.001 or even 0.0005) so they only chase real breakage. But a genuine SRM is usually unmistakable: the p-value is microscopic.

Common culprits: bot or spam filtering applied to one arm but not the other; a redirect or extra load step in the treatment that quietly drops slow users; caching that strands some visitors; logging differences between variants; or assigning users on a key that isn’t stable across sessions. The fix is always to find and repair the cause, then rerun — never to “adjust” the broken numbers.

Novelty and primacy effects

Even with a perfectly clean split, the effect you measure in week one may not be the effect you will live with for months. The reason is that people react to change over time, and that reaction can pull your early numbers in either direction. There are two opposite traps.

The novelty effect: users notice something new and shiny and poke at it just because it is new. A redesigned button gets clicked simply because it looks different. Engagement spikes early, then fades as the novelty wears off. Your week-one result overstates the real, long-run effect — ship on that spike and you will be disappointed later.

The primacy effect (also called inertia) is the mirror image. Long-time users are used to the old way, and a change briefly throws them off — they fumble, they resist, the metric dips. Then they adapt, and the number recovers and maybe climbs above where it started. Your early result now understates the long-run effect — abandon the test too soon and you kill a genuine winner.

How do you tell which is happening? Plot the effect over time, not just the single cumulative number — if the daily effect is still trending up or down, you have not reached the steady state yet. Then segment new users versus returning users: novelty and primacy mostly act on existing users reacting to change, while brand-new users have no “old way” to compare against, so their behavior is a cleaner read of the long-run effect.

Too many metrics: the multiple-comparisons trap

Your test ships with a dashboard of twenty metrics — clicks, revenue, time on page, scroll depth, sign-ups, and on and on. Here is the uncomfortable arithmetic: even if your change did absolutely nothing, run twenty independent checks at a 5% significance level and you expect about one of them to light up by pure chance. Stare at enough metrics and the word “significant” loses its meaning.

We can put a number on it. If each test independently has a 5% false-positive rate, the chance that at least one of m tests fires falsely is one minus the chance that all of them stay quiet:

With α = 0.05 and m = 20 metrics, this is 1 − 0.95^20 ≈ 0.64.

Sixty-four percent. With twenty metrics and a change that truly does nothing, there is about a 64% chance at least one screams “significant” purely by luck. Now slice each metric by country, device, and browser, and the number of comparisons — and false positives — explodes. This is the engine of p-hacking: try enough things, then report whichever one happened to cross the line, as if you had predicted it all along.

There are three honest defenses, and you should use them in order. First, pre-register ONE primary metric — the overall evaluation criterion you chose in the design guide — and judge the test by that single number. Second, treat every other metric as exploratory: it can generate hypotheses for the next test, but it does not get to make this decision. Third, if you genuinely must decide on several metrics at once, correct for the multiplicity.

The simplest correction is the Bonferroni method: if you are testing m metrics and want an overall 5% error rate, require each individual test to clear a stricter bar of α divided by m. It is conservative — it can miss real effects — but it is honest and easy. When there are many metrics, teams often prefer controlling the false discovery rate (FDR), which tolerates a known small fraction of false positives among the things it flags, in exchange for catching more real effects.

Bonferroni: split the error budget evenly across all m comparisons.

A results-readout checklist

Put it all together as a routine you run every time, in order. The order matters: sanity-check that the test even worked before you interpret it, then read the primary metric honestly, and only then let secondary metrics whisper ideas for next time.

1. SRM first. Check the actual traffic split with a chi-square test. If it fails, stop and debug — do not interpret a single number until the split is clean.
2. Did you follow the plan? Confirm you ran to the pre-planned sample size and did not stop early at a lucky peek.
3. Read the primary metric’s effect AND its confidence interval — not just the p-value. How big is it, and how uncertain?
4. Practical significance. Is the whole interval above the threshold (the MDE) you set in advance? Significant is not the same as worth it.
5. Effect over time. Plot it; watch for novelty and primacy. Has it stabilized, or is it still trending?
6. Guardrails. Confirm your guardrail metrics did not regress — a conversion win that tanks page speed or revenue is not a win.
7. Secondary metrics. Treat them as exploratory; correct for multiplicity; use them to form hypotheses, not to make this decision.
8. Decide and write it down. Ship, iterate, or abandon — and record the reasoning so the next person (often future-you) can audit it.

Notice that almost every trap in this guide has the same antidote: decide the rules before you see the data — the metric, the sample size, the stopping point, the success threshold. Honesty in experimentation is mostly pre-commitment, plus the humility to read the interval instead of chasing a single asterisk. Get those two habits and you will outperform analysts with far fancier tools.