Value = expected return from here
A policy tells the agent what to do; a value function tells it how good its situation is. The definition is the bridge we built in guide 1: the value of a state is the expected discounted return you will collect if you start there and follow your policy. Value compresses a whole tangled future into a single number per state — exactly the summary an agent needs to compare options.
The state-value function V
The state-value function V_π(s) is the expected return starting from state s and following policy π afterward. High V means 'a good place to be.' A chess position with a winning attack has high value; a position down a queen has low value — regardless of whose specific move it is. Because V depends on π, a strong policy and a weak policy assign different values to the very same state.
A gridworld with cells shaded by their state value as Q-learning estimates expected returns.
The action-value function Q
Often we want finer information: not just 'how good is this state?' but 'how good is taking this action in this state?' That is the action-value function Q_π(s, a) — the expected return after taking action a in state s and following π from then on. Q is the workhorse of value-based RL: if you have Q_π, choosing the best action is trivial — take the a with the largest Q. This is exactly why Q-learning and DQN estimate Q directly.
V and Q are tightly linked: the value of a state is the average of its action-values weighted by how often the policy takes each action. Conversely, Q is one reward plus the discounted V of where you land.
V is the policy-weighted average of its action-values, while Q is one reward plus the discounted value of where you land.
The advantage: how much better than average?
Subtract V from Q and you get the advantage: A(s, a) = Q(s, a) − V(s). It answers a sharper question than Q alone: 'is this action better or worse than what I would normally do in this state?' Positive advantage means 'above the state's baseline — do more of this'; negative means 'below average — do less.' Advantage is the natural learning signal for actor-critic and policy-gradient methods because it strips away how good the state already was and isolates the action's contribution.
The advantage subtracts the state's baseline value from the action-value, isolating how much better an action is than average.
Baselines: subtracting what you already knew
Why subtract V at all? Because of variance. Raw returns are noisy; two runs from the same good state can both look 'high' even when one action was clearly better. The baseline subtraction idea is to compare each return against a reference — typically V(s) — so we react to the difference, not the absolute level. Crucially, subtracting a baseline that does not depend on the action leaves the policy-gradient unbiased while sharply cutting its variance, which is why nearly every modern method uses a baseline (and GAE to estimate advantage well).
- Estimate V(s): the expected return for the state under the current policy.
- Estimate Q(s, a): the expected return for actually taking action a there.
- Advantage = Q(s, a) − V(s): the action's edge over the state's baseline.
- Push the policy toward actions with positive advantage, away from negative.