Reinforcement Learning Theory

generalized advantage estimation (GAE)

Generalized advantage estimation is a tunable way to answer the question every policy-gradient method needs: how much better than average was this action? A one-step temporal-difference estimate is low variance but biased by a possibly-wrong value function; the full Monte Carlo return is unbiased but high variance. GAE interpolates between these two extremes with a single knob.

It forms an exponentially weighted sum of one-step TD residuals along the trajectory. With residual delta_t equal to the reward plus the discounted next value minus the current value, the GAE advantage is the sum over future steps of (gamma lambda) raised to the lag, times that residual. Setting lambda to zero recovers the one-step TD advantage; lambda equal to one recovers the Monte Carlo advantage; values in between trade bias against variance.

GAE is the standard advantage estimator inside PPO and TRPO. The discount gamma and the trace parameter lambda jointly set the effective planning horizon; common defaults are gamma around 0.99 and lambda around 0.95.

\hat{A}^{\mathrm{GAE}(\gamma,\lambda)}_t=\sum_{l=0}^{\infty}(\gamma\lambda)^l\,\delta_{t+l},\qquad \delta_t=r_t+\gamma V(s_{t+1})-V(s_t)

An exponentially weighted sum of TD residuals; lambda slides from one-step TD to the full Monte Carlo return.

Lambda controls bias-variance only of the advantage estimator; gamma changes the actual objective. Tuning them as if interchangeable is a common confusion.

Also called

GAE廣義優勢估計