Score Matching: Learning a Density Without Its Normalizer

Why the gradient is friendlier than the density

Take the log of an energy-based density: log p(x) = −E(x) − log Z. Now differentiate with respect to x. The term log Z is a constant in x, so it vanishes. The score — defined as *s(x) = ∇ₓ log p(x) = −∇ₓ E(x)* — carries all the shape information of the distribution while being completely free of the intractable normalizer. That single line is the reason this whole subfield exists.

\mathbf{s}(\mathbf{x}) \;=\; \nabla_{\mathbf{x}} \log p(\mathbf{x}) \;=\; -\,\nabla_{\mathbf{x}} E(\mathbf{x}) \;-\; \underbrace{\nabla_{\mathbf{x}} \log Z}_{=\,0}

The score is the gradient of the log-density; the normalizer log Z is constant in x and drops out entirely.

The score-matching objective

We want a model score *sθ(x)* to match the true data score. The obvious loss is the expected squared distance *½ 𝔼ₓ ‖sθ(x) − ∇ log p_data(x)‖²*, but we never have ∇ log p_data. Score matching's elegant trick (Hyvärinen, 2005) uses integration by parts to rewrite that loss into one that depends only on the model: the squared norm of its own score plus the trace of its Jacobian. The data score disappears entirely.

J(\theta) \;=\; \mathbb{E}_{p_{\text{data}}}\!\left[\, \tfrac{1}{2}\,\lVert \mathbf{s}_\theta(\mathbf{x}) \rVert^2 \;+\; \operatorname{tr}\!\big(\nabla_{\mathbf{x}}\, \mathbf{s}_\theta(\mathbf{x})\big) \,\right]

Hyvärinen's identity rewrites the intractable target into this objective — only the model's own score and the trace of its Jacobian remain.

# Hyvarinen score matching (per sample x)
s  = score_net(x)                 # model score s_theta(x)
term1 = 0.5 * (s ** 2).sum()
term2 = trace(jacobian(score_net, x))  # sum of d s_i / d x_i
loss  = term1 + term2             # no data score needed

The objective contains only the model's own score and the diagonal of its Jacobian — never the unknown data score or Z.

There is a catch graduate readers should feel immediately: that Jacobian-trace term costs a backward pass per input dimension, which is hopeless for images. So plain score matching is beautiful but impractical at scale, and the field's real progress came from two cheaper estimators.

Two estimators that scale

Sliced score matching: project both scores onto random directions v and match the scalar vᵀs. The trace becomes a cheap Hutchinson estimator vᵀ∇(vᵀs), a single extra backward pass instead of one per dimension.
Denoising score matching (DSM): add Gaussian noise σ to x, then learn to predict the score of the noised density. The target is known in closed form — it is just −(x̃ − x)/σ², the direction back to the clean point — so the loss is a plain denoising regression with no Jacobian at all.

DSM is the pivotal idea. It says: *learning the score of a noised distribution is identical to learning to denoise.* Hold that sentence — in the next guide it becomes the entire training objective of diffusion models, with the noise level σ promoted from a fixed nuisance to a swept-over time axis.

Sampling from a score: Langevin dynamics

Knowing the score lets you sample even without the density. Langevin dynamics walks uphill along *sθ(x)* while injecting calibrated Gaussian noise; its stationary distribution is exactly p. This is the score-based cousin of classical Markov chain Monte Carlo samplers like Gibbs sampling, and at the parameter level stochastic gradient Langevin dynamics uses the very same recipe to sample posteriors.