advantage function

The advantage function measures not how good an action is in absolute terms, but how much better or worse it is than your policy's average behaviour in that state. Take action a in state s; the advantage is its action value minus the state's value. A positive advantage means this move beats my usual play here; a negative one means I would have done better sticking to my normal habit.

It is defined as the action value minus the state value, and by construction it averages to zero over the policy's own actions. This subtraction is what makes it valuable: it strips away the part of the value that depends only on the state, leaving a cleaner signal about the action itself. Policy-gradient methods lean on it heavily because using advantage instead of raw return sharply reduces the noise in their updates.