policy gradient
/ POL-uh-see GRAY-dee-unt /
Policy-gradient methods take a refreshingly direct approach: instead of laboriously estimating how good each action is and then reading off a strategy, they adjust the strategy itself. The policy is a set of dials (the weights of a neural network) that turn a situation into action probabilities. Policy gradient watches whether episodes turn out well or badly, then turns the dials to make the good behavior more likely and the bad behavior less likely — straightforward reward chasing, applied to the policy directly.
The mechanics are like sculpting by reinforcement. The agent plays out a whole episode, tallies the reward, and then for every action it took, asks: did things go better than expected? If so, push that action's probability up a touch; if worse, push it down. 'Gradient' just means the calculus that tells you which direction to nudge each dial. Run this over many episodes and the policy drifts toward behavior that earns more.
The big payoff is flexibility. Because the policy can output any kind of action — including smooth, continuous ones like a steering angle or a joint torque — policy gradients handle problems where Q-methods choke on infinitely many choices. They also naturally produce stochastic policies, useful for bluffing and exploring. The honest cost is that the raw learning signal is very noisy: judging single actions by an entire episode's outcome makes training jittery and sample-hungry, which is exactly why refinements like actor-critic and PPO exist.
Training a cart to balance a pole: the policy outputs a probability of pushing left vs right. The agent plays an episode; if the pole stays up for a long time (high reward), every push it made during that episode gets made a little more likely next time. After thousands of episodes, the balancing behavior emerges on its own.
Make actions from good episodes more likely, bad ones less likely — adjusting the policy directly, no Q-table needed.
Policy gradients shine where action choices are continuous (robot control), where value-based methods struggle. The price is high variance: because a whole episode's reward is used to judge individual actions, the learning signal is noisy, so these methods often need many samples and careful tuning to train stably.