variance of policy gradients

If you have ever watched a REINFORCE learning curve jitter up and down, you have met this problem first-hand. The policy gradient is an average over sampled trajectories, and each sample is a wild guess: the same policy can earn a great return one episode and a terrible one the next, purely by chance. Average a handful of such noisy numbers and your estimated direction is itself noisy.

Several forces compound the noise. The return G_t aggregates many random rewards, so its spread grows with the horizon. The score factor ∇_θ log π_θ can be large, especially for confident or near-deterministic policies. Long episodes assign the same whole-trajectory return to every action, blurring credit assignment. And the estimator's variance scales roughly inversely with the number of sampled trajectories, so small batches are punishing.

Understanding the sources tells you the cures directly: shorten the effective horizon (discounting, reward-to-go), centre the weights (baselines, advantages), borrow a critic to replace the Monte Carlo return, and average over more or more diverse samples. Essentially the entire design of modern policy-gradient algorithms is a response to this one number being too large.