log-derivative trick

This is the tiny piece of calculus that makes policy gradients work, and it is worth seeing once on its own. The identity is just the chain rule run backwards: the gradient of a probability density equals the density times the gradient of its logarithm. In symbols, ∇_θ p_θ(x) = p_θ(x) ∇_θ log p_θ(x). Trivial-looking, but it is the hinge of the whole field.

Why is it so useful? When you differentiate an expectation E_{x∼p_θ}[f(x)] you hit a sum (or integral) of f(x) times ∇_θ p_θ(x). That last factor is not an expectation, so you cannot sample it. Substitute the identity and the p_θ(x) folds back into the probability you are averaging over, leaving f(x) ∇_θ log p_θ(x) — which is an expectation again, and therefore samplable.

Every score-function estimator, REINFORCE, and the policy gradient theorem itself is this one substitution applied to the expected return. Recognising it lets you derive new estimators on the spot, and it also makes clear the trick's limit: it ignores any path by which f depends on θ directly, which is where reparameterisation and stochastic computation graphs come in.