mutual information
If I tell you the value of X, how much does that reduce your uncertainty about Y? Mutual information answers exactly this: it measures, in bits, how much knowing one random variable tells you about another. It is the information-theoretic cousin of correlation, but far more general — it detects ANY kind of statistical dependence between two variables, not just a straight-line relationship.
There are two equivalent readings, and both are illuminating. As a reduction in uncertainty: I(X; Y) = H(Y) - H(Y given X), the entropy of Y minus the entropy of Y that remains once X is known — the surprise about Y that learning X removes. By symmetry it equals H(X) - H(X given Y), so the information X gives about Y equals the information Y gives about X. As a divergence: I(X; Y) = KL of the joint distribution p(x, y) from the product of marginals p(x) p(y) — it measures how far X and Y are from being independent, since independence is exactly when the joint factors into the product. Either way, the formula is I(X; Y) = sum over x, y of p(x, y) log[ p(x, y) / (p(x) p(y)) ]. Mutual information is always greater than or equal to 0, and it equals 0 if and only if X and Y are independent.
This makes it a sharp tool wherever dependence matters: feature selection in machine learning (keep the inputs most informative about the target), measuring channel capacity in communication, registering medical images, and quantifying how much a neuron's spikes encode about a stimulus. Its great advantage over the correlation coefficient is honesty about nonlinearity — two variables can have zero correlation yet large mutual information, for instance Y = X^2 with X symmetric around zero, where the link is perfect but curved. The caveat: I = 0 means genuine statistical independence (stronger than zero correlation), but estimating mutual information reliably from finite data, especially for continuous variables, is genuinely hard and prone to bias.
Let X be a fair coin (heads/tails) and let Y always equal X (a perfect copy). Then knowing X removes ALL uncertainty about Y, so I(X; Y) = H(Y) = 1 bit. If instead Y is an independent coin flip, learning X tells you nothing about Y and I(X; Y) = 0. Real dependence lands in between, and unlike correlation it catches curved relationships too.
Mutual information is the uncertainty about Y that knowing X removes; zero means independence.
Mutual information is zero exactly when two variables are independent, so unlike correlation it detects nonlinear dependence (e.g. Y = X^2) — but estimating it accurately from finite data, especially for continuous variables, is hard and biased.