Shannon entropy
/ SHAN-un /
How much are you surprised, on average, when you learn the outcome of a random event? A coin you already know lands heads carries no surprise; a fair die carries some; a lottery draw carries a lot. Shannon entropy measures exactly this average surprise — equivalently, the average amount of information you gain from observing the outcome, and equivalently again, the irreducible number of yes/no questions needed on average to pin the outcome down.
Build it from one idea: a less probable outcome is more surprising. Quantify the surprise of an outcome with probability p as log(1/p) = -log p (rare events, big surprise; a sure event, p = 1, gives zero surprise). The entropy of a distribution is just the average surprise: H(X) = -sum over x of p(x) log p(x). Use log base 2 and the unit is bits. A fair coin has H = -[0.5 log2 0.5 + 0.5 log2 0.5] = 1 bit — one yes/no question's worth. A fair die has log2 6 about 2.58 bits. Entropy is largest when the distribution is uniform (maximum uncertainty, all outcomes equally likely) and zero when one outcome is certain (no uncertainty at all). Adding more equally-likely outcomes raises it.
This single number is the foundation of information theory and shows up everywhere probability meets computing. Shannon's source coding theorem makes it concrete: H(X) is the minimum average number of bits needed to encode outcomes of X — you cannot compress below entropy without losing information, which is why a near-uniform file barely compresses while a lopsided one shrinks a lot. In machine learning, minimizing cross-entropy loss trains classifiers; in physics, the same formula (with a different constant) is thermodynamic entropy. The honest caveat: entropy measures uncertainty, not value or meaning — a string of pure random noise has maximal entropy yet carries no useful message, so 'high information' in Shannon's sense is not the same as 'useful'.
Weather forecast for two cities. City A is sunny 90 percent of days: its outcome is nearly certain, so its entropy is low, only about 0.47 bits — knowing the forecast tells you little new. City B is sunny exactly half the days: maximum uncertainty for two outcomes, entropy exactly 1 bit — each day's weather is a genuine surprise carrying a full bit of information.
Entropy is average surprise, maximized by a uniform distribution and zero for a sure thing.
Entropy measures uncertainty, not usefulness: pure random noise has the maximum possible entropy yet carries no meaningful message, so high Shannon information is not the same as valuable information.