the look-elsewhere effect

If you buy one lottery ticket, your chance of winning is tiny. But if you buy a thousand tickets, your chance that at least one of them wins is much larger — even though each single ticket is still a long shot. The look-elsewhere effect is the same idea applied to bump hunting: when you search for a new-particle peak across many possible masses, the chance that some random fluctuation somewhere will fake a peak is far higher than the chance of a fluke at any one particular mass. It is the statistical penalty for not having decided in advance exactly where to look.

Here is the precise problem. Suppose at one specific mass a fluctuation as large as you saw has only a one-in-a-thousand chance of happening by accident. That sounds impressive. But if you scanned a hundred independent mass values looking for a bump anywhere, then the chance that at least one of those hundred shows a one-in-a-thousand fluctuation is roughly one in ten — not nearly so rare. Physicists distinguish the local significance (computed as if you had only looked at the one place where the bump appeared) from the global significance (corrected for the fact that you looked everywhere). The ratio between the two probabilities is called the trials factor.

The look-elsewhere effect is why a striking-looking bump at an unexpected mass is treated with deep suspicion until the global significance is worked out. Many a promising hint — a few-sigma local excess — has shrunk to insignificance once the look-elsewhere correction was applied, because the experiment had been scanning a wide mass range. The honest takeaway is that significance is not a property of the bump alone; it depends on how hard and how widely you searched. A pre-registered search at a single predicted mass needs no such correction, which is one reason theoretical predictions that pin down a mass in advance are so valuable.