generalized advantage estimation (GAE)

GAE solves the central tension in estimating an advantage. You could use a one-step temporal-difference error: low variance, but biased by the critic's mistakes. Or you could use the full Monte Carlo return: unbiased, but extremely noisy. GAE gives you a single dial, λ between 0 and 1, that smoothly blends every horizon in between, letting you pick your own bias-variance trade-off.

It works by taking an exponentially weighted average of the TD residuals δ_t = r_t + γV(s_{t+1}) − V(s_t) over future steps, with weight (γλ)^l on the residual l steps ahead. At λ=0 it collapses to the one-step TD advantage (low variance, biased); at λ=1 it becomes the full Monte Carlo advantage (unbiased, high variance). Values around 0.95 are a common sweet spot in practice.

GAE became standard because it is cheap — just a backward pass over a rollout — and because it pairs perfectly with batched on-policy methods. It is the default advantage estimator inside most PPO and A2C implementations. Its honesty caveat: for any λ below 1 it still inherits whatever systematic error the critic V has.