reparameterization trick

To train a VAE we must back-propagate gradients through a step that draws a random sample z from the encoder's distribution. But randomness is a dead end for gradients — you cannot ask "how would the random draw have changed if I nudged the mean?", because the draw is, well, random. The reparameterization trick is a clever rewrite that moves the randomness out of the gradient's path: instead of sampling z directly from a distribution whose parameters we are learning, we sample a fixed, parameter-free noise and then deterministically transform it using those parameters.

Precisely, suppose the encoder outputs mean μ and standard deviation σ for a Gaussian latent, so z is distributed as N(μ, σ²). Rather than sampling z from that distribution, we sample standard-normal noise ε from N(0, I) — which does not depend on any parameter — and set z = μ + σ ⊙ ε (elementwise multiply-and-add). This z has exactly the right distribution, but it is now a deterministic, differentiable function of μ and σ, with ε an external input. Gradients of the loss with respect to μ and σ (and hence the encoder weights) flow straight through the addition and multiplication; the only stochastic node, ε, has no parameters to differentiate.

This converts a high-variance estimator (the score-function or REINFORCE gradient, which works but is noisy) into a low-variance pathwise gradient, which is why VAEs train stably with ordinary stochastic gradient descent. The trick applies to any distribution that can be written as a deterministic transform of parameter-free noise (the location–scale family, and more generally any "reparameterizable" distribution). For discrete latents it does not directly apply — there you use relaxations such as the Gumbel-Softmax / Concrete distribution, or fall back to score-function estimators.