score-function estimator

You often want the gradient of an average of something that depends on random samples — for example, the expected reward when the action is drawn at random from your policy. The catch is that the randomness itself depends on the parameters, so you cannot just differentiate the thing inside the average. The score-function estimator is the standard escape hatch: it turns the gradient of an expectation into an expectation of a gradient that you can sample.

The recipe is to multiply the sampled value f(x) by the gradient of the log-probability of the sample, ∇_θ log p_θ(x), and average over samples. This is an unbiased estimate of ∇_θ E[f(x)] for any well-behaved f, even one that is not differentiable or whose form you do not know. In RL, f is the return and p_θ is the policy, which is precisely how the policy gradient theorem is computed in practice.

Its great strength is generality — it needs nothing from f but samples and rewards. Its great weakness is variance, because the log-probability factor can be large and is uncorrelated with f. That is why score-function estimates are almost always paired with baselines and other variance-reduction tricks before they are usable.