Ornstein–Uhlenbeck noise

A deterministic policy like DDPG's outputs one action and never explores on its own, so exploration must be added by perturbing that action with noise. The naive choice is fresh independent random noise each step, but in physical control that produces useless jitter: the agent twitches back and forth and barely moves anywhere, because each push is cancelled by the next. Ornstein–Uhlenbeck noise solves this by being temporally correlated — today's noise remembers yesterday's.

OU noise comes from a simple stochastic process that drifts back toward zero while being nudged randomly, so it wanders in smooth, persistent excursions rather than flickering. Applied to a robot, that means a sustained gentle push in some direction for a while, then another — exploration that actually covers ground, like leaning on the throttle rather than tapping it. The process has a couple of knobs: how strongly it pulls back to zero, and how large the random kicks are.

OU noise was a signature ingredient of the original DDPG paper and is well-suited to systems with momentum and inertia. Interestingly, later work found that for many tasks plain uncorrelated Gaussian noise works just as well, and TD3 and SAC mostly dropped OU in favour of simpler noise or built-in stochasticity. It remains worth knowing as a clean example of why correlated exploration can matter in control.