score-based generative model

A score-based generative model attacks generation from a different-looking angle that turns out to be the same mountain. Instead of modeling the probability of an image directly (which requires an intractable normalizing constant), it models the score: the gradient of the log-probability with respect to the image. Intuitively the score is an arrow at every point in image space pointing 'uphill' toward more plausible images. If you know which way is uphill everywhere, you can start from a random image and keep nudging in the uphill direction (with a little random jitter) until you arrive at a realistic sample.

The score is the vector field s_theta(x) = grad_x log p(x). It is attractive because it sidesteps the normalizing constant Z of a probability density — the gradient of log p does not depend on Z. The network is trained by score matching, and the sampling rule is Langevin dynamics: repeatedly take a step in the direction of the estimated score and add a calibrated amount of Gaussian noise. The catch is that the score is ill-defined and hard to estimate in regions where data is sparse (almost everywhere in high dimensions). Song and Ermon's key fix was to estimate the score at many noise levels at once (noise-conditional score networks), starting sampling at high noise where the score is easy and annealing down to low noise where it is sharp.

This 'estimate the score across noise levels and anneal' recipe is, step for step, the same thing diffusion models do — and the connection was made exact by the stochastic-differential-equation (SDE) framework, which casts both as continuous-time processes. DDPM corresponds to the variance-preserving SDE; the score model corresponds to the variance-exploding SDE. The payoff of this unification is enormous: it lets you borrow numerical ODE/SDE solvers as fast samplers, gives a principled probability-flow ODE for exact likelihoods, and explains why 'predict the noise' and 'predict the score' are the same network up to a known rescaling.