action saturation

Action saturation is when a policy keeps slamming its actions against the allowed limits — full throttle, full steer, maximum torque — instead of using the smooth middle range. Picture a driver who only ever floors the gas or stomps the brake, never feathering either. Sometimes that is genuinely optimal (so-called bang-bang control), but more often it is a symptom: the policy has been pushed to its bounds by an over-eager critic, and the rich continuous control space has collapsed into a crude on/off one.

It often arises hand-in-hand with the tanh squashing used to enforce bounds. Out near ±1 the tanh is flat, so its gradient is tiny; once a policy drifts into saturation, learning signals barely flow back and it can get stuck there, jammed against the wall. The result is jerky, energy-wasteful, hardware-stressing behaviour in robots, and a value function that has learned almost nothing about the interior of the action space.

Diagnosing it means watching the distribution of chosen actions — a healthy continuous policy spends real time in the interior. Remedies include penalising large or rapidly changing actions in the reward, scaling the action range sensibly, and the entropy bonus of maximum-entropy RL, which actively discourages the policy from collapsing onto the extremes.