JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Beyond REINFORCE: Deterministic, Off-Policy, and Modern Policy Gradients

The frontier of the field: deterministic gradients for continuous control, off-policy reuse of data, the computation-graph view that unifies it all, and how TRPO/PPO/SAC grew from these roots.

Deterministic policy gradients

The score-function estimator works by jiggling action probabilities — but in high-dimensional continuous control that randomness is itself a huge variance source. The deterministic policy gradient takes a different path: let the actor output a single action μ(s; θ), and push it using the gradient of the critic's Q-value with respect to the action, ∇θ μ · ∇a Q. The signal flows through the critic by the chain rule, with no sampling over actions.

\nabla_\theta J(\theta) = \mathbb{E}_{s \sim \rho^\mu}\left[ \nabla_\theta \mu_\theta(s)\, \nabla_a Q^\mu(s,a)\big|_{a=\mu_\theta(s)} \right]

The deterministic policy gradient: push the policy along the critic's action-gradient ∇ₐQ instead of reweighting random actions.

This is the heart of DDPG and its more stable successor TD3, which add twin critics and delayed updates to curb the value overestimation that deterministic actors are prone to. They are off-policy by design, learning from a replay buffer like the value-based methods.

Off-policy policy gradients

Plain policy gradients are on-policy: each batch must come from the current policy, then is discarded. That is wasteful. Off-policy policy gradients reuse data gathered by an older or different behavior policy, correcting the mismatch with importance sampling ratios π_new/π_old. The reward of doing this is far better sample efficiency; the price is that those ratios can explode when the policies drift apart, reigniting the variance problem.

L^{\mathrm{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[ \min\left( r_t(\theta)\,\hat{A}_t,\ \mathrm{clip}\left(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon\right)\hat{A}_t \right) \right], \quad r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)}

PPO's clipped surrogate keeps the importance ratio r_t(θ) near 1, bounding how far one off-policy batch can move the policy.

The stochastic computation graph view

There is a unifying lens for everything above: stochastic computation graphs. Any objective that mixes deterministic nodes with sampling nodes can be differentiated by two estimators. Where you cannot push gradients through a sample, use the score-function (REINFORCE-style) estimator. Where you can — by writing the sample as a deterministic function of noise — use the reparameterization estimator, which is usually far lower variance.

This is exactly why a squashed-Gaussian policy can be trained by backpropagation: sample ε from a fixed noise distribution, then set a = tanh(μ(s) + σ(s)·ε). The gradient flows straight through. Deterministic policy gradients are just the limiting case where the noise vanishes.

Reparameterization lets the gradient flow straight through the sampling node — the same backpropagated descent you can roll down here.

Interactive gradient descent rolling down a loss surface, illustrating how backpropagated gradients move parameters toward lower loss.

Where the field is today

These ideas converged into the algorithms you will actually deploy. TRPO makes the largest policy step that stays within a trusted KL region, guaranteeing monotonic-ish improvement. PPO keeps that safety with a simple clipped objective and a first-order optimizer — it is the default for most on-policy work and the engine behind RLHF. Soft Actor-Critic (SAC) marries the reparameterized off-policy gradient with a maximum-entropy objective, giving the best sample efficiency on continuous control.

RLHF is where these policy gradients meet language models: a reward model from human preferences tunes the policy with PPO.

Diagram of RLHF: human preference comparisons train a reward model, which provides the signal to fine-tune the policy.

  1. Discrete actions, simple and robust? Start with A2C/PPO and an entropy bonus.
  2. Continuous control, want sample efficiency? Reach for SAC or TD3 with a replay buffer.
  3. Fine-tuning a language model from human feedback? PPO with a KL penalty to a reference policy is the proven path.
  4. Whatever you pick, monitor the KL/clip fraction and entropy — they are the dashboard lights of a healthy policy gradient run.