statistical significance
/ stuh-TIS-tih-kul sig-NIF-ih-kunss /
Your new model scores 91 percent and the old one scored 90. Is it genuinely better, or did it just get lucky on the particular test examples you happened to use? Statistical significance is the discipline of telling a real difference apart from random luck. It asks: if there were truly no difference, how often would mere chance produce a gap this big or bigger?
That "how often" is captured by the p-value. A small p-value (the conventional cutoff is 0.05) means a gap this large would rarely happen by pure chance, so the difference is called "statistically significant" — probably real. A large p-value means the gap is well within what luck alone could cook up, so you can't claim a real improvement. Crucially, this hinges on sample size: a tiny test set makes everything look like noise, while a huge one can flag even trivial differences as significant.
Here is the caveat that trips up almost everyone, and it's vital. Statistical significance is not the same as importance. With enough data, a microscopically small improvement — say 0.01 percent — can be highly significant yet utterly meaningless in practice. And significance says nothing about whether your experiment was well-designed; garbage in, significant garbage out. Worse, if you test many variations and report only the ones that came out significant ("p-hacking"), you will manufacture false discoveries. Significance is a guard against fooling yourself with luck — but it is not a stamp of truth or value.
Model A beats Model B by 1 point on 50 test items — flip a few items and the lead vanishes, so p = 0.4: not significant. The same 1-point gap measured over 50,000 items, where it holds steadily, gives p = 0.001: significant. Same gap, opposite verdict — sample size decides.
The same difference can be noise on small data and solid on large data.
Two enduring confusions to resist. First, a p-value is not the probability your hypothesis is true — it's the probability of seeing data this extreme if there were no real effect. Second, "significant" ≠ "large" or "important"; always report the effect size alongside the p-value, and be honest about how many things you tested.