variance reduction (policy gradients)

The defining practical problem of policy gradients is noise: each gradient estimate is built from random episodes, so it scatters wildly around the true direction and learning crawls. Variance reduction is the umbrella name for the family of tricks that quiet this noise without throwing the gradient off course — that is, without introducing bias. Mastering them is most of what separates a textbook REINFORCE from a working agent.

The standard toolkit, roughly in order of impact: subtract a baseline (ideally a learned V(s)), so only above-average actions are reinforced; use reward-to-go so each action is credited only with the rewards it could have influenced; learn a critic and use advantages instead of full returns; blend horizons with GAE; average over many parallel actors to shrink the estimator's variance; and clip or normalise gradients to keep updates well-scaled.

There is no free lunch: some reducers, like a learned critic or GAE with λ<1, trade a little bias for a lot less variance, which is usually a good deal but must be watched. The art is choosing reducers whose remaining bias you can tolerate, and recognising that almost all of an algorithm's design — baseline, advantage, batching — is really variance reduction in disguise.