evidence lower bound (ELBO)

Latent-variable models say that data arise from hidden causes, but computing the data's likelihood means summing over every possible hidden cause, an integral that is usually intractable. The evidence lower bound is a computable quantity that always sits below this true log-likelihood, so maximizing the bound pushes the likelihood up while staying tractable. It is the engine behind variational autoencoders and, by extension, much of modern probabilistic generative modeling.

The bound is derived by introducing an approximate posterior over the latent and applying Jensen's inequality, or equivalently by noting that the log-likelihood equals the ELBO plus the Kullback-Leibler divergence from the approximate to the true posterior. Since that divergence is non-negative, the ELBO lower-bounds the evidence, and the gap closes exactly when the approximate posterior equals the true one. The bound decomposes neatly into a reconstruction term, how well latents explain the data, minus a regularizer pulling the approximate posterior toward the prior, which is the form trained end to end with the reparameterization trick.

Maximizing the ELBO does double duty: it fits the generative model and simultaneously trains the inference network, but because the bound is loose by exactly the posterior gap, a model can score well on the ELBO yet have an inaccurate posterior, which motivates tighter bounds.