deep deterministic policy gradient (DDPG)

DDPG is the first algorithm most people meet for continuous control, and it is best understood as Q-learning fitted with an extra helper. The problem is that you cannot compute max_a Q(s,a) over real-valued actions. DDPG's answer is to train a second network — an actor — whose only job is to output the action it believes maximises Q for a given state. The critic learns Q the usual way; the actor learns to climb it. Together they recover the spirit of Q-learning in a continuous world.

The actor is deterministic: for a state it returns one action, not a distribution, written a = μ_θ(s). Its gradient is delightfully direct — push the actor's output in whatever direction raises the critic's value, which is the chain rule ∇_θ Q(s, μ_θ(s)) flowing through the critic into the actor. DDPG is off-policy, so it trains from a replay buffer, and it borrows target networks from DQN to keep the moving Q-target stable. Exploration is added by hand, by perturbing the actor's action with noise.

DDPG was a breakthrough but is notoriously brittle: it overestimates Q, is sensitive to hyperparameters, and can collapse mid-training. Almost everything that came after — TD3 and SAC especially — is a list of fixes for DDPG's specific failure modes. It remains the cleanest place to learn the actor-critic-for-continuous-actions pattern before adding the stabilisers.