JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Taming Variance: Advantages, GAE, and Entropy

Policy gradients are unbiased but noisy. Meet the toolkit — baselines as control variates, the bias–variance dial of GAE, and entropy bonuses — that turns a shaky estimate into a trainable one.

Why the gradient is so noisy

The score-function estimator is unbiased, but its variance can be enormous. Returns are sums of many random rewards; multiply them by log-probability gradients and the spread compounds. High variance means each update points in a slightly random direction, so you must use tiny learning rates and many samples — exactly the bottleneck that makes policy gradients data-hungry.

\nabla_\theta J(\theta)=\mathbb{E}\!\left[\sum_{t}\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,G_t\right]

The score-function estimator: each log-probability gradient is scaled by the full return G_t, so noisy returns inject large variance.

Variance reduction is therefore not a luxury; it is what separates a method that works from one that merely converges in theory. The good news: we can lower variance a lot without introducing bias, and trade a little bias for a lot more stability when we choose to.

Baselines as control variates

Seen through a statistical lens, a baseline is a control variate: a quantity with known (zero) expected effect on the gradient that is correlated with the noisy term, so subtracting it cancels noise. The variance-minimizing baseline is roughly the expected return of the state — its value. Replacing the raw return with return-minus-value gives the advantage A(s, a): how much better action a was than the state's average.

GAE: a dial between bias and variance

How should we estimate the advantage? A full Monte-Carlo return is unbiased but high-variance; a one-step bootstrap (reward plus discounted next-state value) is low-variance but biased by the critic's errors. Generalized Advantage Estimation (GAE) blends all horizons with an exponential weight λ, giving a single knob: λ near 0 trusts the critic (low variance, more bias), λ near 1 trusts observed returns (low bias, more variance).

GAE's λ is a dial: small λ leans low-variance but biased, large λ leans unbiased but high-variance.

Bias-variance trade-off curves: total error as the sum of bias and variance against estimator complexity.

Note that the discount factor γ and λ play distinct roles: γ defines the problem (how much the future matters), while λ is purely an estimator choice for trading bias against variance. In practice γ ≈ 0.99 and λ ≈ 0.95 are reliable starting points across many tasks.

Entropy: keep the policy curious

A second failure mode is premature collapse: the policy becomes nearly deterministic too early, stops exploring, and locks into a mediocre habit. Entropy regularization counters this by adding a small bonus for keeping the action distribution spread out. It pushes against the exploration–exploitation tilt toward exploitation, holding the door open for better moves to be discovered.

\mathcal{L}(\theta)=-\,\mathbb{E}\!\left[A_t\,\log\pi_\theta(a_t\mid s_t)\right]-\beta\,\mathbb{E}\!\left[\mathcal{H}\big(\pi_\theta(\cdot\mid s_t)\big)\right]

The entropy-regularized loss adds a bonus β·H that rewards a curious, higher-entropy policy.

  1. Estimate advantages with GAE using your critic and a chosen λ.
  2. Normalize advantages (subtract mean, divide by std) within each batch — a cheap, strong variance reducer.
  3. Form the loss: −(advantage · logπ) − β·entropy, with a small entropy coefficient β.
  4. Anneal β downward over training so the policy sharpens once it has explored enough.