deterministic policy gradient (DPG)

Standard policy gradients assume a stochastic policy you can sample from and take the log-probability of. But in continuous control — steering a car, torquing a joint — it is often cleaner to output one specific action, a deterministic function μ_θ(s). The deterministic policy gradient is the form of the update built for exactly that case, and it changes the maths in an illuminating way.

Instead of weighting a score by a return, the DPG follows the chain rule straight through the critic: it is the expectation of ∇_θ μ_θ(s) times ∇_a Q(s,a) evaluated at a = μ_θ(s). In words, nudge the action the actor outputs in the direction that the critic says increases value, then push the parameters to produce that nudged action. Because there is no action distribution to integrate over, the estimator often has much lower variance than its stochastic cousin.

The catch is exploration: a deterministic actor will not explore on its own, so you must inject noise into the actions during training, which makes DPG inherently off-policy. This is exactly the recipe behind DDPG and TD3, the workhorse deep methods for continuous control, where a differentiable critic supplies the action gradient.