target policy smoothing

Target policy smoothing is one of TD3's three fixes, and it cures a sneaky failure of deterministic actor-critics. When the actor proposes a single best action and the critic happens to have a sharp, spurious spike there — an over-optimistic bump that is just noise in the function approximation — the actor will eagerly steer the policy straight onto that fake peak. Smoothing makes the critic refuse to trust razor-thin spikes by forcing it to be smooth around each action.

The mechanism is small: when computing the learning target, add a little clipped random noise to the target action before evaluating the critic, a' = μ(s') + clip(ε). Because nearby actions should have similar value, this averages the critic over a small neighbourhood, so a lone spike gets blurred away and only genuinely broad regions of high value survive into the target. It acts as a regulariser on the value function — a built-in assumption that good actions come in smooth basins, not needle points.

Conceptually this is a way to bake in the prior that the world is locally smooth: if an action is good, slightly different actions should be good too. That assumption is almost always true in physical control, where a millimetre of difference in joint angle should not flip an outcome, which is exactly why the trick is so effective there.