Jensen's inequality

Here is a question with a surprisingly general answer: if you average some numbers and then square them, do you get the same thing as squaring first and then averaging? Try 1 and 3. Averaging then squaring: (2)^2 = 4. Squaring then averaging: (1 + 9)/2 = 5. They differ, and the second is bigger. Jensen's inequality explains exactly when and which way 'average then transform' and 'transform then average' disagree.

The key word is convex: a function whose graph bows upward, so the curve sits below any chord (line segment) joining two of its points — squaring, the exponential, and -log are convex examples. Jensen's inequality says, for a convex function g, g(E[X]) <= E[g(X)]. Geometrically: the value of g at the average input lies below the average of g's values, because the curve dips beneath its chords. Reverse the inequality for concave functions (those bowing downward, like log or square root): g(E[X]) >= E[g(X)]. A handy memory aid is 'the convex function of the mean is at most the mean of the function.'

This single fact powers a remarkable amount of mathematics. It instantly proves E[X^2] >= (E[X])^2, which is why variance is non-negative. It gives the arithmetic-mean / geometric-mean inequality (via the concave log). In information theory it underpins the non-negativity of Kullback-Leibler divergence and entropy bounds. The honest caveat: the direction depends entirely on convexity versus concavity, and equality holds only if g is linear over the range of X, or X is a constant — otherwise the gap is strict and is a genuine, measurable error you make by swapping the order of averaging and transforming.