Hypothesis tests & p-values

The courtroom analogy: innocent until proven

You changed something — maybe you suspect a coin is weighted, you swapped in a new checkout button, or a friend swears their new study app raised their grades — and the numbers moved a little. The hard question is never “did the number change?” It almost always changes a bit. The hard question is: was that movement real, or just the kind of random wobble you would see even if nothing had truly changed? A hypothesis test is the formal ritual statisticians use to answer exactly that question.

The cleanest way to feel how it works is to picture a courtroom. A defendant is presumed innocent. The prosecution does not have to prove innocence false by magic; it has to present evidence so strong that “innocent” becomes hard to believe. Notice the shape of the reasoning: we never prove guilt with total certainty. We ask a sharper question — would this evidence be surprising if the defendant were actually innocent? If it would be very surprising, we convict.

Two consequences fall out of this analogy, and they trip up almost everyone. First, a test can only reject the null or fail to reject it — it can never “accept” it or prove it true, exactly as a “not guilty” verdict is not the same as “proven innocent.” Absence of strong evidence is not strong evidence of absence. Second, a verdict can be wrong in two different ways: you can convict an innocent person, or you can let a guilty one walk free. Hold on to those two mistakes — we will give them names and probabilities later.

Null and alternative hypotheses

Let’s make it concrete with the simplest possible experiment. You flip a coin 100 times and get 60 heads. A fair coin “should” land on about 50. Is 60 far enough from 50 to accuse the coin of being rigged — or is 60 just luck? Before we compute a single thing, statistics asks you to write down two competing statements about the world.

The first is the null hypothesis, written H-zero: the coin is fair — the true, long-run probability of heads is exactly 0.5. This is always the “no effect, no difference, nothing interesting” claim. The second is the alternative hypothesis, written H-one: the coin is not fair — the true probability is something other than 0.5. The alternative is the interesting claim you would only believe if the data forced you to.

The null pins down one exact number (0.5), which is what lets us compute. The alternative just says “not that.”

One choice to make up front: one-sided or two-sided. If you would only care about the coin favoring heads, your alternative is “probability greater than 0.5” (one-sided). If a coin biased in either direction counts, your alternative is “not equal to 0.5” (two-sided). The honest default is two-sided, because before you looked you had no reason to bet on a particular direction — and you must decide this before seeing the data, not after the result tempts you.

The test statistic and the p-value

Now we need to squeeze all 100 flips into one number that measures how far our result sits from what the null predicts. That number is the test statistic. The trick is to measure the distance not in raw heads, but in units of the ordinary random wobble you would expect anyway — a unit called the standard error.

Recall the idea from sampling: even a perfectly fair coin almost never gives exactly 50 heads in 100 flips; the count scatters around 50 from run to run. For 100 flips, that typical scatter — the standard error of the proportion — works out to the square root of 0.5 times 0.5 divided by 100, which is 0.05, or 5 percentage points. So “one unit of luck” here is about 5 heads-percent.

Our observed 60% sits exactly two standard errors above the 50% a fair coin predicts.

So our result is two standard errors away from the null. Is “two standard errors” a lot? To answer, we finally compute the p-value. Say it carefully in words first: assume the null is exactly true — the coin really is fair — and imagine repeating this 100-flip experiment over and over forever. The p-value is the fraction of those imaginary experiments that would give a result at least as extreme as the one you actually saw (60 or more heads, or 40 or fewer, since we chose two-sided).

In words: how often would pure luck, under the null, match or beat what we saw?

For our coin, a result two or more standard errors out in either direction lands, on a bell-shaped curve, only about 5% of the time — so the p-value is roughly 0.05. A more exact calculation (the binomial test in the snippet below) gives 0.057. Either way, the plain-English reading is the same: if the coin were truly fair, a split this lopsided or more would turn up only about once in every 18 experiments. Uncommon — though, as we are about to see, sitting right on the borderline of the usual cutoff.

from scipy.stats import binomtest
# 60 heads in 100 flips - is the coin fair (p = 0.5)?
result = binomtest(60, n=100, p=0.5, alternative="two-sided")
print(round(result.pvalue, 4))   # -> 0.0569

What a p-value is — and the four things it is NOT

Because the p-value is so easy to compute and so widely quoted, it is also the single most misread number in all of science. So let’s nail the one correct definition and then dismantle the four most common misreadings — this is the honesty core of the whole guide. The one true definition: a p-value is the probability of getting data at least as extreme as yours, computed in a hypothetical world where the null hypothesis is exactly true.

The p-value is the probability of the data given the null — not the probability of the null given the data. Those two are not the same, and swapping them is the original sin of p-values.

1. It is NOT the probability that the null hypothesis is true. You want the probability the coin is fair given your data; the p-value is the reverse conditional — the probability of your data given a fair coin. Flipping the two requires Bayes’ rule and a prior belief. So p = 0.057 does not mean “there is a 5.7% chance the coin is fair.”
2. It is NOT the probability that your result happened “just by chance.” The p-value already assumes that chance — the null — is the entire story; it is computed inside that assumption. A number computed by assuming chance cannot also be the probability that the assumption itself is true.
3. It is NOT a measure of how big or how important the effect is. Collect a huge sample and a tiny, useless difference — a coin biased by half a percentage point — can produce a microscopic p-value. Collect a small sample and a large, genuinely important effect can produce a fat, non-significant p-value. Statistical significance and practical importance are different axes.
4. It is NOT the probability that you are wrong if you reject the null. How often a “significant” finding turns out to be a false alarm depends on how many real effects there were to find in the first place — the base rate — not on the p-value alone. Hunt for effects where almost none exist, and most of your “discoveries” will be wrong even at p below 0.05.

Significance, type I and type II errors

A p-value is a continuous measure of surprise, but at some point you have to act — ship the button or not, recall the coin or not. So before the experiment you pick a threshold called the significance level, written α (the Greek letter alpha), most often set to 0.05. The rule is simple: if the p-value comes in at or below α, you call the result statistically significant and reject the null; otherwise you fail to reject it.

The decision rule. Note that α is set in advance, by you — not read off the data.

Back to the courtroom: because any verdict can be wrong, statisticians name the two ways. A type I error is rejecting a null that was actually true — a false alarm — like shouting “the coin is rigged!” when it was fair all along. The neat part is that you control its long-run rate directly: it is exactly α. Choosing α = 0.05 is a promise that, in a world where the null is true, you will cry wolf only 5% of the time.

The opposite mistake is a type II error: failing to reject a null that was actually false — a miss — the coin really is rigged, but your data did not catch it and you let it walk. Its long-run rate is written β (beta). Unlike α, you do not get to set β directly; it depends on how big the real effect is and how much data you collected.

Two errors, two conditional probabilities — each defined in a different world (null true vs null false).

Statistical power and why underpowered tests mislead

If a type II error is a miss, its mirror image is the good outcome, and it has its own name: statistical power is the probability that your test correctly catches a real effect when one truly exists. Power and the miss rate are two sides of one coin — power equals one minus β.

Power: the chance of (rightly) rejecting the null when the alternative is the truth.

Make it concrete. Suppose a new checkout button really does lift the purchase rate from 8.0% to 8.4% — a real effect, but a small one. If you test only 200 visitors per group, the random wobble between groups is far larger than 0.4 of a percentage point, so even though the lift is genuine, your test will almost always come back with a boring, non-significant p-value. Low power means you systematically miss real wins and wrongly conclude “nothing happened.” The common standard is to design for at least 80% power, i.e. a miss rate β of 0.20 or less.

Four dials turn power up: a bigger true effect (easier to see), less noisy data (a smaller standard deviation), a more lenient α (a lower bar to clear), and — the dial you actually control — a larger sample. That last one is why serious teams calculate the required sample size before running anything; we devote a whole guide to it in the experiments track.

p-hacking and the replication crisis

Remember that the 5% false-alarm rate is per test. Run many tests and the false alarms quietly pile up. If you test 20 independent hypotheses that are all truly null, at α = 0.05 you should expect about one of them to come back “significant” purely by chance. Search 20 colors of jelly bean for a link to acne and one color will almost surely look guilty — and of course that is the one color that ends up in the headline.

With 20 independent true nulls, the chance of at least one false alarm is not 5% but about 64%.

This is the machinery behind p-hacking (also called data dredging): the practice — often not even deliberate — of trying many analyses and reporting only the ones that cross p below 0.05. Testing many outcomes, slicing the data into many subgroups, adding and dropping control variables until something turns significant, or peeking at the results and stopping the instant they look good — every extra attempt is another lottery ticket for a false positive, and reporting only the winners is like a marksman who fires at a wall and then paints the target around the bullet hole.

Two more forces make it worse. Publication bias: journals, conferences, and internal dashboards all prefer exciting, significant results, so the literature fills up with the lucky 5% while the boring, null 95% quietly stay in a drawer — the so-called file-drawer problem. And once a striking result is published, the incentive to re-check it is weak. Put p-hacking, low power, and publication bias together and you get a literature that looks far more certain than the underlying evidence ever justified.

1. Decide your one main outcome, the exact test, and the sample size before you see the data — and where you can, pre-register that plan so you cannot quietly change it later.
2. If you genuinely must test many things, correct for it — use a stricter threshold via a multiple-comparisons adjustment instead of judging each test at a naive 0.05.
3. Do not stop the moment you hit significance. Fix the sample size up front, or use a proper sequential method designed to allow early stopping.
4. Report everything you tried, including the null results, and always show effect sizes with confidence intervals — not just a scattering of significance stars.
5. Trust replication over any single dazzling p-value. One study is a hint; the same effect, found again by other people with fresh data, is knowledge.

A sane checklist for reading a test

You now know the whole ritual — hypotheses, a test statistic, a p-value, a threshold, two errors, and power — and, just as important, how it gets abused. Here is a compact way to read anyone’s “p below 0.05,” including your own, without fooling yourself.

1. What exactly were the hypotheses, and were they fixed in advance? An analysis that chose its question after seeing the data is a red flag, not a finding.
2. What is the effect size, in real units? “Significant” says the effect is probably not exactly zero; it never says the effect is big enough to matter to anyone.
3. Where is the confidence interval? A wide interval that almost reaches zero is weak evidence even when the p-value squeaks under 0.05.
4. Was the study adequately powered? A tiny sample that nonetheless “finds” a large effect should raise suspicion, not excitement — remember the exaggeration trap.
5. How many things were tested to get here? One significant result out of fifty comparisons is exactly what pure noise looks like.
6. Has it replicated? A single study is a lead worth following; an effect found again and again, independently, is something you can build on.

This is the frequentist toolkit — the dominant way statistics is taught and reported. But there is a second, very different way to reason about evidence: start from a prior belief, update it with the data, and read your answer directly as a probability about the world. That is the subject of the next guide, Bayesian thinking. And whenever you want to put a number on how big an effect is, rather than just a yes-or-no verdict, return to confidence intervals — the most decision-friendly tool in this whole track.