statistical significance

You see a bump in your data that looks exactly like the new particle you were hoping for. But random noise also produces bumps, and if you stare at enough data some bump somewhere is inevitable. Statistical significance answers the sharp, sobering question you must ask before celebrating: if there were really nothing there but known background, how unlikely would it be for chance alone to fake a bump at least this large?

You set up a null hypothesis, that there is no signal and only the known background, and compute the p-value: the probability that the background alone would fluctuate to produce an effect as extreme as, or more extreme than, the one observed. A small p-value means such a fluke is improbable. In particle physics the p-value is conventionally translated into a number of sigma using the Gaussian tail: 3 sigma (p ~ 0.0013, about 1 in 740) counts as 'evidence', while 5 sigma (p ~ 3 x 10^-7, about 1 in 3.5 million) is the threshold to claim a 'discovery', the bar the Higgs boson had to clear in 2012. Because a signal's significance grows roughly as S / sqrt(B), it improves as sqrt(N) with more data.

Significance is the agreed guardrail against fooling yourself with noise, made deliberately stringent because experimenters examine so many possible bumps. The honest caveat is deep and often misstated: the p-value is the probability of the data given no signal, not the probability that there is no signal, and those are genuinely different (the latter needs a prior). A 5 sigma result is not '99.99997% certain to be real'; it is a statement about how rarely background would fake it. And significance by itself says nothing about systematic errors or the look-elsewhere effect, the fact that searching many bins inflates the chance of some fluke somewhere, both of which must be accounted for separately.