Diffusion Models: From DDPM to the SDE and the ODE

The denoising idea

Denoising diffusion probabilistic models (DDPM) define a fixed forward process that gradually adds Gaussian noise over T steps until the data is indistinguishable from pure noise. Because each step is Gaussian and the schedule is fixed, the marginal at any step t has a closed form — you can jump straight to x_t from *x_0* in one shot. The model learns only the reverse chain: given a noisy x_t, predict the slightly less noisy *x_{t−1}*.

This is the trade that broke the generative trilemma from Guide 1. Instead of asking one network to leap from noise to a photorealistic image in a single bound, we ask it for a sequence of tiny, locally-Gaussian corrections. Each step is an easy regression; the difficulty is amortized over hundreds of steps.

Training is just denoising score matching, swept over time

Here the previous guide pays off. The DDPM training loss reduces to predicting the noise ε that was added at a randomly sampled timestep — and predicting ε is exactly denoising score matching, because the optimal noise predictor is proportional to the score *∇ log p_t(x_t)* of the noised marginal. So a DDPM is a score-matching model that estimates the score at every noise level at once, which is why it is also called score-based diffusion. The annealing that single-level Langevin lacked is now built in.

# DDPM training step
t   = randint(1, T)               # random noise level
eps = randn_like(x0)
xt  = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps  # closed-form
loss = mse(eps_net(xt, t), eps)   # predict the injected noise

One forward jump to x_t, one ε-prediction. Simple, stable, and equivalent to score matching at noise level t.

The continuous view: forward and reverse SDEs

Let the number of steps go to infinity and the chain becomes a stochastic differential equation (SDE): a forward SDE that smoothly drives data into noise. Anderson's classical result says this forward SDE has a corresponding reverse-time SDE that turns noise back into data — and the only unknown in that reverse equation is the score *∇ log p_t(x)*, the very thing we trained. DDPM, score-based models, and a whole zoo of noise schedules become special cases of one continuous framework.

dx=\left[f(x,t)-g(t)^{2}\,\nabla_{x}\log p_t(x)\right]dt+g(t)\,d\bar{w}

The reverse-time SDE: subtract the score-scaled drift and inject noise to drive noise back toward data.

The deterministic twin: probability-flow ODE and DDIM

Every diffusion SDE has a deterministic partner, the probability-flow ODE, whose trajectories carry exactly the same marginal distributions p_t at every time — but with no injected noise. Solving an ODE is much cheaper than simulating an SDE, you can use high-order solvers, and because the map from noise to data is now deterministic and invertible, you get a well-defined latent encoding of any sample (useful for editing and exact likelihood).

\frac{dx}{dt}=f(x,t)-\tfrac{1}{2}\,g(t)^{2}\,\nabla_{x}\log p_t(x)

The probability-flow ODE: the deterministic twin with the same marginals p_t but half the score term and no injected noise.

DDIM is the discrete, non-Markovian sampler that realizes this deterministic path. It shares DDPM's training but reinterprets sampling so you can skip large chunks of the schedule, dropping a thousand steps to a few dozen with little quality loss. DDIM is the practical bridge between 'slow stochastic DDPM' and 'fast deterministic ODE'.

Steering the score: classifier-free guidance

To make a diffusion model follow a prompt, you condition the score on a label or text c. Classifier-free guidance trains one network for both the conditional score *∇ log p_t(x | c)* and the unconditional *∇ log p_t(x)* (by randomly dropping c), then at sampling time extrapolates along their difference: a guidance weight pushes samples toward more strongly class-consistent regions. It is the workhorse behind modern text-to-image systems and, with a VAE front end, latent diffusion.

\tilde{\epsilon}_\theta(x_t,c)=(1+w)\,\epsilon_\theta(x_t,c)-w\,\epsilon_\theta(x_t)

Classifier-free guidance extrapolates the conditional prediction away from the unconditional one by a weight w.

The honest caveat: guidance trades diversity for fidelity. Crank the weight up and samples look crisper and more on-prompt but collapse toward a few prototypes, oversaturate, and lose the tails of the distribution. Choosing the weight is a genuine bias–variance dial, not a free quality knob — a recurring theme you should distrust whenever a demo looks suspiciously clean.