baseline subtraction

Imagine grading students not by raw score but by how far above or below the class average each one landed. Baseline subtraction does the same for actions: instead of judging an action by its raw return, you subtract a reference value — a baseline — and judge it by the difference. The choice still gets pushed up if it beat the baseline and down if it fell short, but the numbers are smaller and steadier.

The key fact is that subtracting any baseline that does not depend on the action leaves the expected policy-gradient update unchanged — it adds no bias — yet it can dramatically cut the variance of that update. The most natural baseline is the state value, and using it turns the raw return into the advantage. This is why algorithms estimate a value function purely as a baseline: not to act on directly, but to make the learning signal far less noisy.