concentration of measure

Average enough independent things and a near-miracle happens: the result barely varies. Flip a fair coin ten times and the head-count is all over the place; flip it ten million times and the fraction of heads is glued to one-half with astonishing reliability. This phenomenon — that a function of many independent contributions is overwhelmingly likely to sit very close to its mean — is called concentration of measure. It is the deep reason randomness becomes predictable in bulk.

Concretely, the spread of an average of n independent terms shrinks: its standard deviation falls like 1 / sqrt(n), and (under boundedness or light tails) the probability of deviating by even a fixed amount falls exponentially in n. The collection of theorems that make this precise — Markov, Chebyshev, Chernoff, Hoeffding, Bernstein, Azuma, McDiarmid — are the 'concentration inequalities'. They all express the same message at sharpening levels of precision: a high-dimensional random object, or a sum of many small independent pieces, hugs its expected value, and the chance of a meaningful deviation is tiny. The deeper modern form notices that ANY well-behaved (Lipschitz) function on a high-dimensional space is nearly constant — the randomness 'concentrates' onto a thin shell.

Concentration is the quiet workhorse behind much of statistics, machine learning, randomized algorithms, and statistical physics. It is why polls with a few thousand respondents pin down a nation's opinion, why averaging noisy measurements cleans them up, and why generalization is possible at all — the training-set performance of a model concentrates around its true performance. The honest caveat: concentration needs the contributions to be (nearly) independent and not too heavy-tailed. Strong dependence or heavy tails (infinite variance, like the Cauchy) break it — there a single rare term can dominate the whole average, and it never settles down.