kl divergence

KL divergence measures how different one probability distribution is from another — roughly, the extra "surprise" you suffer if you believe distribution Q but reality follows P. In a VAE it appears as a regularizer: a penalty that pushes the encoder's per-image latent distribution to look like the simple prior (a standard Gaussian). Without it, the encoder could scatter each image's code into its own faraway, tightly-packed pocket, leaving the space between pockets empty so that random samples decode to nonsense. The KL term herds all the codes into one shared, smooth, samplable region.

Precisely, KL(Q || P) equals the expectation over z drawn from Q of log Q(z) minus log P(z); it is always non-negative, equal to zero only when Q = P, and it is not symmetric (KL(Q||P) generally differs from KL(P||Q)). In the VAE, Q is q_φ(z|x) = N(μ(x), σ²(x)) and P is p(z) = N(0, I), and for two Gaussians the KL has a closed form: for each latent dimension it equals one-half of (μ² + σ² − 1 − log σ²), summed over dimensions. This is exactly the second term of the ELBO; it costs nothing when μ = 0 and σ = 1 (the code distribution equals the prior) and grows as the code drifts away or becomes over-confident (very small σ).

The KL term controls an information bottleneck: each nat of KL is a nat of information the code is allowed to carry about x. If the decoder is powerful enough to reconstruct x without using z (for example an autoregressive decoder), the optimizer may drive the KL to zero — the encoder outputs the prior for every image and the latent becomes useless. This failure is posterior collapse. Mitigations include KL annealing or warm-up (ramp the KL weight up slowly), a free-bits floor (do not penalize KL below a small per-dimension budget), weakening the decoder, or the β-VAE knob (β < 1 to discourage collapse, β > 1 to encourage disentanglement at the risk of collapse).