advantage-based update

An advantage update is the natural endpoint of the baseline idea. Instead of asking 'was this whole episode good?', it asks the sharper question 'was this action better than the average action available in this state?'. That comparison is the advantage A(s,a) = Q(s,a) − V(s): positive means the action beat the state's default, negative means it underperformed.

In a policy gradient you replace the raw return with the advantage, so the update becomes ∇_θ log π_θ(a|s) times A(s,a). Because V(s) acts as an ideal baseline, the advantage is centred near zero, which dramatically lowers variance while keeping the gradient unbiased. The critic estimates V (or Q), the actor follows the advantage-weighted gradient, and the two are trained together — the core loop of A2C, A3C, GAE-based methods, and PPO.

The catch is that the advantage is now an estimate, so the critic's errors leak in as bias. Methods differ mainly in how they estimate the advantage: one-step temporal-difference is low-variance but biased, full Monte Carlo is unbiased but noisy, and generalized advantage estimation slides between the two with a single knob.