policy gradient theorem

Imagine you want a robot to walk and the only thing you can touch is the set of dials (parameters) inside its decision rule. The policy gradient theorem tells you which way to turn those dials: raise the probability of actions that were followed by good returns, lower the probability of ones followed by bad. The lovely part is that you never need a model of how the world reacts to your dials — you only need to know how your own action probabilities depend on them.

Formally, take the objective J(θ) to be the expected return of policy π_θ. The theorem says its gradient equals an expectation over trajectories of the score ∇_θ log π_θ(a|s) weighted by the action-value Q^π(s,a). The environment's transition probabilities vanish from the derivative because they do not depend on θ, so you can estimate the whole gradient just by sampling episodes and averaging — which is exactly what makes the method practical.

This one result underlies every algorithm in this field: REINFORCE, actor-critic, A2C, and PPO are all ways of estimating or stabilising it. Its Achilles' heel is variance — the Q-weights swing wildly from episode to episode, so raw estimates are noisy and learning is slow. Baselines, advantages, and critics all exist to cut that variance without biasing the direction.