significance level
/ sig-NIF-i-kans LEV-el /
A smoke alarm has to make a choice: be so touchy that it shrieks at the faintest wisp, or so relaxed that it sleeps through real fires. Where you set its trigger is a deliberate decision about how often you are willing to be wrongly alarmed. The significance level is the same kind of dial for a statistical test.
The significance level, written as alpha, is the probability you are willing to accept of raising a false alarm — of rejecting the null hypothesis when it is actually true. It is chosen in advance, most commonly at 0.05, meaning you accept a 5 percent chance of declaring a difference that is really just noise. A result that crosses this threshold is called statistically significant.
It matters because it forces you to fix your standard of evidence before you peek at the answer, which keeps the test honest. The caveat is twofold: lowering alpha to avoid false alarms makes you more likely to miss genuine effects, and statistical significance is not the same as practical importance — a difference can clear the threshold yet be far too small to matter in the real world.
Two methods are compared with a t-test at a significance level of 0.05. The test returns a probability of 0.03 that the observed difference is mere chance, which is below 0.05, so the difference is declared significant.
A pre-set threshold for how much chance you will tolerate.
The confidence level is just one minus the significance level: a 0.05 significance level corresponds to 95 percent confidence. Both must be chosen before the data are examined, never tuned afterward to get the answer you wanted.