Huber loss (in RL)

When training a value network, you minimise the gap between its prediction and a target. Squared error is the obvious choice, but in RL targets are bootstrapped and noisy, and the occasional huge error — a surprising reward, a bad bootstrap — gets squared into an enormous gradient that can jolt the weights and destabilise training. The Huber loss is the calmer alternative: it behaves like squared error for small errors but switches to plain absolute error once the error grows past a threshold.

Below the threshold it is smooth and quadratic, so gradients shrink gracefully near the target; above it the loss grows only linearly, so its gradient saturates at a constant and a single outlier can no longer dominate the update. It is, in effect, a built-in form of gradient clipping for the loss. DQN used the Huber loss precisely for this robustness, and it pairs naturally with reward clipping as part of the recipe that keeps deep value learning stable. The same loss reappears, smoothing the pinball loss, inside QR-DQN.