JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Policy Gradients: Optimizing Behavior Directly

Instead of learning values and acting greedily, tune the knobs of the policy itself with gradient ascent — and meet the trick that makes it possible.

Two ways to get a good policy

Most early reinforcement learning learns a value — how good each state or action is — and then acts greedily. Policy gradient methods take a different route: they treat the policy itself as a function with tunable parameters (say, the weights of a neural network) and directly nudge those parameters in the direction that increases reward. There is no detour through a value table.

The agent–environment loop policy gradients optimize: the policy acts, the environment returns reward, and we tune θ toward higher return.

Diagram of the reinforcement-learning loop: the agent takes an action, the environment returns the next state and a reward, and the cycle repeats.

This is powerful precisely because the policy can be anything differentiable. It can output a probability over a handful of discrete moves, or the mean and spread of a continuous steering angle. That makes policy gradients the natural tool for continuous action spaces where a greedy 'argmax over actions' is intractable.

The objective: expected return as a dial

Write the policy as π(a | s; θ), where θ are the parameters. Our goal is the expected return J(θ): if we run this policy for whole episodes and average the total discounted reward, how high can it go? Policy gradient methods perform gradient ascent on J(θ) — they compute ∇θ J and step θ a little in that direction, again and again.

J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[\, \sum_{t=0}^{T} \gamma^{t} r_t \right]

The objective J(θ): expected return averaged over whole trajectories the policy itself generates.

The catch is obvious once you stare at it: J(θ) is an expectation over trajectories, and the trajectories themselves are produced by the very policy we are differentiating. The reward function may be unknown and non-differentiable. So how do you take a gradient through an environment you cannot differentiate?

The log-derivative trick

The answer is a small algebraic identity called the log-derivative trick. For any distribution p(x; θ), we have ∇θ p = p · ∇θ log p. Substituting this into the gradient of an expectation turns ∇θ E[f] into E[ f · ∇θ log p(x; θ) ]. The gradient hops off the reward (which we cannot differentiate) and lands entirely on the log-probability of our own choices (which we can).

This produces the score-function estimator (also called the likelihood-ratio estimator). It is breathtakingly general: you only need to be able to sample from the policy and evaluate ∇θ log π(a | s; θ). The environment can stay a black box.

The policy gradient theorem

Apply that trick to the return objective and you get the policy gradient theorem: ∇θ J(θ) = E[ Σt ∇θ log π(at | st; θ) · Rt ], where Rt is the return that followed time t. The remarkable part is what is missing — there is no gradient of the environment's transition dynamics. Changing the policy changes which states you visit, yet that effect cancels out cleanly, leaving an expectation you can estimate from sampled experience alone.

\nabla_\theta J(\theta) = \mathbb{E}\!\left[\, \sum_{t} \nabla_\theta \log \pi(a_t \mid s_t; \theta)\, R_t \right]

The policy gradient theorem: push up the log-probability of each action, weighted by the return that followed it.

That single equation is the engine room of this whole track. Everything that follows — REINFORCE, baselines, actor-critic, TRPO, PPO, SAC — is an answer to one practical question: the theorem gives an unbiased gradient estimate, but a wildly noisy one. How do we make it usable?