score matching

Suppose you want to model a probability density, but the normalizing constant, the integral that makes it sum to one, is impossible to compute. Score matching sidesteps this entirely. Instead of matching the density itself, it matches the density's shape, specifically the gradient of the log-density, which practitioners call the score. Because the gradient of the log eliminates any constant factor, the intractable normalizer simply disappears, and you can fit the model without ever evaluating it.

Hyvärinen's 2005 objective minimizes the expected squared distance between the model score and the unknown data score. That data score is not available, but integration by parts converts the loss into a tractable form involving the divergence of the model score plus half its squared norm, requiring only the model and its own derivatives. Computing the divergence, a Hessian trace, is expensive in high dimension, so scalable variants dominate practice: sliced score matching estimates the trace with random projections, and denoising score matching replaces the data with data convolved with Gaussian noise, for which the target score is known in closed form.

Denoising score matching is the workhorse behind diffusion models, because the score of the noised data at each level is exactly what the reverse process needs. The same normalizer-free property makes score matching a natural training principle for energy-based models, where the partition function is otherwise the central obstacle.