reparameterization trick

To train anything by gradient descent you must differentiate through it, but how do you differentiate through a random sampling step? The reparameterization trick rewrites draw z from a distribution that depends on the parameters as draw fixed noise from a parameter-free distribution, then transform it deterministically. The randomness is pushed into the noise, which has no gradient, and the parameters now live in a smooth function you can backpropagate through.

For a Gaussian variational distribution with mean mu and standard deviation sigma you write the sample as mu plus sigma times standard-normal noise; the gradient of an expectation with respect to mu and sigma then becomes an expectation of the gradient of the function evaluated at that sample, which a single Monte-Carlo draw estimates with far lower variance than the score-function or REINFORCE estimator. Kingma and Welling made this the engine of the variational autoencoder, letting the ELBO be optimized end-to-end by ordinary backpropagation.

It is the reason gradient-based variational inference scales, and it generalizes via flows and the reparameterizable, location-scale families, with inverse-CDF or implicit reparameterization covering others. The limitation is that it requires a differentiable, reparameterizable sampling path: discrete latents need surrogates like the Gumbel-softmax or straight-through estimators, or fall back on score-function estimators, because you cannot push a gradient through a hard categorical draw.