expectation and variance

Expectation and variance are the two summary numbers that capture the gist of an uncertain quantity: roughly what to expect, and how much to brace for surprises. Expectation, also called the expected value or mean, is the long-run average — the value you would settle on if you repeated the random process a zillion times and averaged the results. It is each possible outcome weighted by how likely it is.

Variance measures spread: how far outcomes typically wander from that average. A small variance means results huddle tightly near the mean (predictable); a large variance means they fling themselves all over the place (volatile). Because variance is built from squared distances, its units are awkward, so people often take its square root to get the standard deviation, which lives in the same units as the data and is easier to picture.

These two ideas are everywhere in machine learning. A model's expected error guides what we try to minimize during training. The bias-variance tradeoff — a cornerstone of why models overfit or underfit — is named directly after variance. And the central trick of stochastic gradient descent, estimating a slope from a small random batch, works precisely because that estimate has the right expectation even if any single batch is noisy. One honest caveat worth holding onto: the expected value can be a number that never actually occurs — the expected roll of a die is 3.5, a face that does not exist — so the average is a useful fiction, not a prediction of any single outcome.