Policy Gradient Methods

actor-critic architecture

Picture two cooperating learners. The actor is the policy — it decides what to do. The critic is a value estimator — it watches and scores how the actor is doing. The actor improves by following the critic's feedback, and the critic improves by checking its predictions against what actually happened. Together they combine the strengths of policy gradients (works for continuous and stochastic actions) with those of value learning (low-variance, step-by-step feedback).

Mechanically, the critic learns V(s) or Q(s,a) using temporal-difference updates, and supplies the actor with an advantage signal in place of the noisy Monte Carlo return. The actor then takes a policy-gradient step weighted by that advantage. Crucially the critic gives feedback at every step rather than only at episode's end, so learning is faster and far smoother than plain REINFORCE.

The trade-off is that the critic introduces bias and a second thing to tune — its learning rate, its architecture, how its errors propagate. If the critic is badly fit, it can mislead the actor and the whole system diverges. Almost every modern policy-gradient algorithm — A2C, A3C, DDPG, SAC, PPO — is some careful variation on this architecture.

advantage = reward + gamma * V(next_state) - V(state)
actor_loss  = -log_prob(action) * advantage.detach()
critic_loss = advantage.pow(2)

The critic scores the step; the actor follows the resulting advantage.

Also called

actor-critic