Maps of the Territory: Four Families of Generative Models

What a generative model actually promises

A discriminative model learns *p(y | x)*. A generative model is more ambitious: it tries to capture the whole data distribution *p(x)* — the manifold of plausible images, sentences, or molecules — so that you can sample new points, score how likely a given point is, and often recover a latent code that explains it. These three capabilities (sample, evaluate density, infer latents) rarely come together for free, and which ones you get cleanly is exactly what separates the model families.

The honest starting point of all of generative modeling is one painful fact: writing down a flexible density p(x) = p̃(x) / Z is easy, but the normalizing constant Z = ∫ p̃(x) dx is a high-dimensional integral we usually cannot compute. Every family in this guide is, at heart, a different strategy for never having to evaluate Z.

p(x) = \frac{\tilde{p}(x)}{Z}, \qquad Z = \int \tilde{p}(x)\, dx

The whole fight in one line: a flexible density is easy to write up to the intractable normalizer Z.

Family I — variational bounds (the VAE lineage)

Latent-variable models write p(x) = ∫ p(x | z) p(z) dz. That marginal is intractable, so instead of maximizing log *p(x)* directly we maximize a tractable lower bound on it — the evidence lower bound (ELBO). The gap between the ELBO and the true log-likelihood is exactly the KL divergence from the approximate posterior *q(z | x)* to the true posterior, so tightening the bound and learning a good encoder are the same job. This is the engine inside the variational autoencoder.

\log p(x) \;\ge\; \mathbb{E}_{q(z\mid x)}\!\left[\log p(x\mid z)\right] - D_{\mathrm{KL}}\!\left(q(z\mid x)\,\|\,p(z)\right) \;\equiv\; \mathcal{L}_{\mathrm{ELBO}}

The evidence lower bound the VAE lineage maximizes: a reconstruction term minus a KL pull toward the prior.

The bound can be slack, and a loose bound biases the model toward posteriors that are easy to match rather than ones that are true. The importance-weighted autoencoder (IWAE) tightens it by averaging several latent samples under an importance weight: with k samples the bound provably approaches log *p(x)* as k → ∞, buying fidelity with compute. Understanding why a tighter bound is not always a better encoder gradient is a classic graduate-level subtlety here.

# k-sample IWAE bound (single x)
z = q.sample(k)                  # k latents from encoder
logw = p_xz(x, z) + p_z(z) - q_zx(z, x)
bound = logsumexp(logw) - log(k) # tighter than mean(logw)

The IWAE bound is a log-mean-exp of importance weights; the plain ELBO is the mean of the log-weights, and Jensen's inequality separates them.

Family II — exact likelihood by construction (flows)

Normalizing flows refuse to approximate the likelihood at all. They build *p(x)* as an invertible, differentiable transform of a simple base density (a Gaussian), and read off the exact log-density via the change-of-variables formula — you pay only the log-determinant of the transform's Jacobian. The whole architectural game is designing layers that are expressive yet keep that Jacobian cheap (triangular, so its determinant is a product of diagonals).

\log p(x) = \log p_z\!\big(f(x)\big) + \log\left|\det \frac{\partial f}{\partial x}\right|

How flows read off exact log-likelihood: the base density at f(x) plus the log-determinant of the Jacobian.

The bargain: flows give you exact log-likelihood and exact sampling, which makes them honest yardsticks for density estimation. The price is the invertibility constraint — every layer must be a bijection of matching input/output dimension, which limits the kinds of bottlenecks and compression a VAE enjoys freely.

Families III & IV — energy and implicit

Energy-based models (EBMs) embrace the unnormalized density head-on: *p(x) ∝ exp(−E(x))* for any neural network energy E. This is maximally flexible — no architectural straitjacket at all — but now both training and sampling must dodge Z. We will spend the next guide on exactly that escape, because it leads straight to diffusion.

Implicit models give up on density entirely. A generative adversarial network (GAN) only needs a generator that maps noise to samples, trained against a critic. The Wasserstein GAN replaces the original Jensen–Shannon objective with the Earth-Mover (Wasserstein-1) distance, whose dual gives the critic a meaningful, non-saturating gradient even when generator and data distributions barely overlap — the key fix for the brittle, mode-collapsing training of early GANs.

The generative trilemma — and the road ahead

Stand back and a pattern appears. You want three things — fast sampling, high sample quality, and broad mode coverage (tied to likelihood) — and historically you could pick about two. GANs gave fast, sharp samples but dropped modes; flows and VAEs covered modes and gave likelihoods but lagged on quality; EBMs were flexible but slow. Diffusion models, the centerpiece of this track, broke the deadlock by trading a single hard problem for many easy denoising steps.

Guide 2 — score matching: learn ∇ log p without ever touching Z; the bridge from EBMs to diffusion.
Guide 3 — diffusion: DDPM, the SDE view, the probability-flow ODE, and guidance.
Guide 4 — continuous-time transport: normalizing flows → continuous flows → flow matching.
Guide 5 — the frontier: few-step, distilled, and discrete-state generation.