relative entropy as a rate function
Relative entropy — the Kullback-Leibler divergence H(nu | mu) — is the rate function in Sanov's theorem, and recognising it as a large-deviation cost is what gives this otherwise information-theoretic quantity its sharpest probabilistic meaning. It answers: among all distributions, how exponentially expensive is it for n iid mu-samples to produce an empirical histogram resembling nu? The answer is e^(-n H(nu|mu)), so relative entropy is literally the per-sample exponential price of pretending mu is nu.
Relative entropy of nu with respect to mu is H(nu | mu) = integral of log(dnu/dmu) dnu when nu is absolutely continuous with respect to mu, and +infinity otherwise. It is non-negative (Gibbs' inequality), zero if and only if nu = mu, jointly convex in (nu, mu), and lower semicontinuous with compact sublevel sets in the weak topology — exactly the properties of a good rate function, which is why Sanov's theorem produces a good LDP. It is not symmetric and is not a metric: H(nu|mu) generally differs from H(mu|nu), reflecting the asymmetric question 'how rare is nu under mu-sampling', which is genuinely directional.
Its role pervades the subject. The Cramer rate function I(x) = sup_theta(theta x - log M(theta)) equals the minimal relative entropy H(nu|mu) over all nu with mean x, the variational form that makes the exponential tilt the optimiser. The Donsker-Varadhan variational formula expresses H(nu|mu) = sup over bounded f of (integral f dnu - log integral e^f dmu), the exact dual of Varadhan's lemma. And the Gibbs conditioning principle says a constrained system relaxes to the relative-entropy minimiser, which is the statistical-mechanics statement that systems maximise entropy subject to constraints. Relative entropy is the thread linking large deviations, information theory and equilibrium statistical mechanics.
For two Bernoulli laws, nu = Bernoulli(p) and mu = Bernoulli(q), H(nu|mu) = p log(p/q) + (1-p) log((1-p)/(1-q)). With q = 1/2 this becomes p log(2p) + (1-p) log(2(1-p)) — exactly the coin-tossing rate function from the LDP entry, confirming that Cramer's coin rate is a special case of Sanov's relative entropy.
KL divergence is the exponential cost of mistaking mu for nu.
Relative entropy is asymmetric and is not a distance: H(nu|mu) is not H(mu|nu), and it violates the triangle inequality. Treating KL as a metric leads to wrong intuitions; it is a directional cost, with mu the reference (sampling) law.