evidence lower bound

The evidence lower bound is the quantity a VAE actually maximizes, because the thing we truly want to maximize — the probability the model assigns to the real data, log p(x), called the "evidence" — is too hard to compute. The ELBO is a tractable floor that always sits below the evidence: push the floor up and you push the evidence up with it (or at least never overshoot it). It conveniently splits into two readable pieces: "reconstruct the data well" minus "keep the latent code close to the prior".

For any approximate posterior q_φ(z|x), one can show that log p(x) equals the ELBO plus KL(q_φ(z|x) || p(z|x)), where the final term is the KL divergence between the encoder's approximate posterior and the true (intractable) posterior. Because a KL divergence is always non-negative, log p(x) ≥ ELBO — hence "lower bound". The gap is exactly that KL, so the bound is tight precisely when the encoder matches the true posterior. The ELBO itself equals E over q_φ(z|x) of log p_θ(x|z), minus KL(q_φ(z|x) || p(z)).

The first term, the expected reconstruction log-likelihood, encodes x to a code distribution, samples a code, decodes, and measures how probable the original x is — high when reconstructions are faithful. The second term, KL(q_φ(z|x) || p(z)), is a regularizer pulling each input's code distribution toward the prior p(z) = N(0, I); it both organizes the latent space for sampling and acts as an information bottleneck (it costs "nats" to deviate from the prior). Maximizing the ELBO thus simultaneously trains the decoder (parameters θ) to reconstruct and the encoder (parameters φ) to infer good, prior-consistent codes.