reparameterization policy gradient

When a policy is stochastic, computing its gradient means differentiating through an act of sampling, and there are two ways to do it. The score-function way (REINFORCE) treats the sampler as a black box and is general but noisy. The reparameterisation way rewrites the sampling so the randomness is pulled out into a fixed source, leaving a deterministic, differentiable path from the parameters to the action — and that path gives much lower-variance gradients, which is gold in continuous control.

The trick is concrete. Instead of sampling an action directly from a Gaussian whose mean and spread depend on θ, draw a standard noise ε ∼ N(0,1) once, then set a = μ_θ(s) + σ_θ(s)·ε. Now ε carries all the randomness and does not depend on θ, so you can backpropagate the critic's gradient straight through a, the same way you would through any deterministic network. The expectation's gradient becomes the gradient of a sample, and the noise has been factored cleanly out of the derivative.

This is why SAC's actor is trained with reparameterisation rather than the score function: with continuous actions and a critic to push against, the pathwise gradient is dramatically less noisy and learning is faster and steadier. The cost is that it needs the action to be a differentiable function of the noise, which works for Gaussians and squashed Gaussians but not for discrete choices.