JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

TD3: Taming Overestimation

Three small, sober tricks turn fragile DDPG into a reliable workhorse. Each one targets a specific way the critic lies.

The lie at the heart of DDPG

Why does DDPG's critic become over-optimistic? The target uses the value of the actor's chosen action, and the actor was trained to choose whatever action the critic rates highest. Any random error where the critic happens to over-rate an action gets selected and copied into the target — a feedback loop that amplifies noise upward. This is overestimation under function approximation.

It is the continuous cousin of maximization bias: taking a max (or, here, an argmax via the actor) over noisy estimates systematically picks the ones that got lucky upward. Twin Delayed DDPG (TD3) is essentially DDPG plus three targeted defenses against exactly this.

Maximization bias in a gridworld: taking a max over noisy Q-estimates systematically picks lucky-high values — the discrete cousin of DDPG's overestimation.

Interactive Q-learning gridworld where an agent learns state-action values.

Trick 1 — clipped double-Q

TD3 trains two critics with different initial weights and, when forming the learning target, uses the smaller of their two value estimates. This clipped double-Q move is deliberately pessimistic: if both critics agree a value is high, it probably is; if only one is excited, the minimum vetoes the over-optimism.

y = r + \gamma \min_{i=1,2} Q_{\theta'_i}\!\left(s', \tilde{a}\right)

The clipped double-Q target: TD3 builds its learning target from the smaller of its two critics' value estimates.

Trick 2 — target policy smoothing

A deterministic critic can develop a sharp, narrow spike of value at one exact action — a spike that is almost certainly an artifact, not a real feature of the world. Target policy smoothing blurs it out: when computing the target, add a little clipped noise to the target action so the value is averaged over a small neighborhood of actions. If a spike cannot survive a tiny jiggle, the actor should not be allowed to chase it.

\tilde{a} = \pi_{\phi'}(s') + \epsilon, \qquad \epsilon \sim \mathrm{clip}\!\left(\mathcal{N}(0,\sigma),\, -c,\, c\right)

Target policy smoothing: add small clipped noise to the target action so similar actions are forced to have similar values.

Think of it as a regularizer that says: similar actions should have similar values. It makes the action-value surface that the actor climbs gentler and more trustworthy.

Trick 3 — delayed policy updates

If the actor chases a critic that is still wrong, it locks onto bad actions and feeds even worse data back in. TD3 therefore updates the actor (and the target networks) less often than the critics — typically once for every two critic updates. Let the critic settle toward the truth first; only then move the actor. That is the 'delayed' in the name.

# forming the target (the heart of TD3)
a2 = actor_target(s2)
a2 = a2 + clip(noise, -c, +c)           # trick 2: target smoothing
a2 = clip(a2, low, high)
q_target = min(critic1_t(s2,a2), critic2_t(s2,a2))   # trick 1: clipped double-Q
y = r + gamma * q_target

update critic1, critic2 toward y        # every step
if step % 2 == 0:                        # trick 3: delayed
    update actor along dQ1/da * da/dw
    soft_update(all targets)
TD3 = DDPG + min of two critics + smoothed target action + actor updated half as often.

The payoff

None of these tricks is clever in isolation — they are sober engineering against a known failure mode. Together they turn the fragile DDPG into a method that trains reliably across MuJoCo with far less seed-to-seed drama. TD3 remains a strong, simple baseline; if you want one deterministic continuous-control algorithm in your toolbox, this is it.