DDPG: A Deterministic Actor Meets a Critic

The core idea in one sentence

Deep Deterministic Policy Gradient (DDPG) takes the actor-critic picture from the last guide and asks a clean question: if the critic Q(s,a) tells me how good an action is, why not just follow the slope of Q to improve the actor? Move the actor's output in the direction that makes Q go up.

The actor here is a deterministic policy: one state in, exactly one action out, no randomness. That is what lets us differentiate cleanly — there is a single number coming out of the actor that we can nudge.

The deterministic policy gradient

Concretely, the actor update uses the chain rule. The deterministic policy gradient says: take the gradient of Q with respect to the action (how would the value change if I nudged the torque?), then push that through the gradient of the actor with respect to its weights. Compose them and you get the direction to move every weight so that the action the actor chooses scores higher under the critic.

\nabla_{\theta^\mu} J \approx \mathbb{E}_{s\sim\mathcal{D}}\Big[\,\nabla_a Q(s,a)\big|_{a=\mu(s)}\;\nabla_{\theta^\mu}\mu(s)\,\Big]

The deterministic policy gradient: chain the critic's gradient in the action with the actor's gradient in its parameters.

Off-policy machinery: replay and target networks

DDPG is off-policy, which means it can learn from data gathered by older versions of itself. Every transition (state, action, reward, next state) is dropped into a replay buffer, and training samples random mini-batches from it. This reuses each hard-won robot interaction many times — crucial when each step costs real motor wear.

Training a critic on its own moving predictions is unstable, so DDPG keeps slow-moving copies — target networks — of both actor and critic to compute the learning target. Instead of copying weights abruptly, DDPG nudges the targets a tiny fraction toward the live networks each step (a soft update, often called Polyak averaging). Slow targets, stable learning.

\theta^{-} \leftarrow \tau\,\theta + (1-\tau)\,\theta^{-}, \qquad \tau \ll 1

Soft target update: the target networks creep toward the live ones by a tiny fraction tau each step instead of being copied wholesale.

Exploring without randomness in the policy

A deterministic actor always returns the same action for a state, so on its own it would never try anything new. DDPG adds exploration noise on top of the action when collecting data. The original paper used Ornstein-Uhlenbeck (OU) noise — temporally correlated noise that drifts smoothly rather than jittering — because correlated exploration is gentler on systems with momentum, like a robot leg.

for each step:
    a = actor(s) + noise          # explore
    a = clip(a, low, high)        # respect motor limits
    s2, r, done = env.step(a)
    buffer.add(s, a, r, s2, done)

    # learn from a random mini-batch
    batch = buffer.sample()
    y = r + gamma * critic_target(s2, actor_target(s2))   # target
    update critic to match Q(s, a) -> y                    # regression
    update actor along  dQ/da * da/dweights               # policy gradient
    soft_update(targets)

DDPG's loop: act with noise, store, then regress the critic and push the actor uphill on Q.

When DDPG shines and when it cracks

DDPG was a landmark — the first method to learn good policies from raw action-value estimates across many MuJoCo tasks. But practitioners quickly learned it is brittle: results swing wildly with the random seed, it is fussy about hyperparameters, and on hard tasks the critic tends to become wildly over-optimistic, dragging the actor toward actions that look great on paper and fail in reality.

That over-optimism — plus the action saturation problem from guide one — is exactly what the next method, TD3, was built to fix.