the strong law of large numbers

The weak law says the sample mean is PROBABLY close to the true mean at each large n. The strong law makes a bolder promise about the entire infinite run: if you could watch the running average forever along a single realization of the experiment, it would actually converge to the true mean and stay there — not just hover nearby, but settle for good.

Precisely: if X_1, X_2, ... are independent and identically distributed with finite mean mu, then the running average X-bar_n = (X_1 + ... + X_n)/n converges almost surely to mu. 'Almost surely' means: the set of infinite sequences for which the average fails to converge to mu has probability zero. Fix one universe — one infinite stream of coin flips, say — and in all but a negligible collection of universes, the fraction of heads truly tends to one half as a genuine limit, in the ordinary calculus sense. Kolmogorov's theorem is sharp: a finite mean is exactly what is needed (and sufficient) for this.

Why does 'strong' beat 'weak'? Because almost sure convergence is strictly stronger than convergence in probability. The weak law allows the average to wander outside any band infinitely often (just with vanishing probability at each n); the strong law forbids that for almost every path — eventually the average enters every band around mu and never leaves. This is the law that justifies the frequentist meaning of probability itself: 'probability is long-run frequency' is a theorem, the strong law applied to indicator variables, not merely a definition.