twin delayed DDPG (TD3)

TD3 is DDPG with three targeted repairs, and it is the algorithm to reach for when DDPG keeps falling over. Its name spells out the first two: twin critics and delayed updates. The diagnosis behind it is that DDPG systematically overestimates how good actions are — the actor learns to exploit any state where the critic is too optimistic, and that error compounds until learning derails. TD3 attacks that overestimation head-on.

The three fixes are: keep two independent critics and use the smaller of their two value estimates when forming the learning target, so the optimistic one cannot run away (clipped double-Q); update the actor and the target networks less often than the critics, letting the value estimate settle before the policy chases it (the delay); and add a little noise to the action used in the target, so the critic is forced to be smooth and cannot latch onto a sharp, fake peak (target policy smoothing). None is large alone; together they transform stability.

TD3 keeps DDPG's deterministic actor and off-policy replay, so it is a drop-in upgrade rather than a new paradigm, and it is often the strongest deterministic method on continuous benchmarks. Its main rival is SAC, which instead adds stochasticity and entropy; the practical choice between them is usually empirical, task by task.