reward clipping

Different Atari games hand out wildly different point values — one awards single points, another thousands per hit. A single network with one learning rate cannot comfortably learn value targets that span such different scales; the big-reward games would dominate the gradients and destabilise training. Reward clipping is the blunt fix used by the original DQN: squash every reward to a fixed range, typically minus one, zero, or plus one, by taking only its sign.

The win is that one set of hyperparameters then works across all games, which is exactly what a general agent needs. The loss is that the agent goes blind to magnitude: a tiny reward and a huge jackpot look identical, so it can no longer prefer the bigger prize when both are reachable. This is a real limitation, and later distributional and value-rescaling methods, such as the invertible transform used in R2D2, were invented partly to keep rewards on a common scale without throwing away how big they are.