gradient clipping
/ GRAY-dee-ent KLIP-ing /
Gradient clipping is a safety belt for training. Sometimes the gradient — the signal telling each weight how to change — comes back enormous, and a single update lurches the model wildly, undoing all its progress in one bad step. Clipping simply caps how big that update is allowed to be: if the gradient is too large, you shrink it back down to a safe size before applying it, keeping the direction but taming the magnitude.
The most common form is clipping by norm. You measure the overall size (the norm) of the whole gradient; if it exceeds a chosen threshold, you scale the entire gradient down proportionally so its size equals the threshold. This preserves which way the model wants to move while preventing a runaway leap. A simpler variant clips each individual value to a maximum, but norm clipping is usually preferred because it doesn't distort the direction.
This matters most where exploding gradients are common: recurrent networks trained through time, and very deep networks. In those settings a rare huge gradient can spike the loss to infinity or produce "NaN" values that ruin the run. Clipping is cheap insurance against that catastrophe. It is not, however, a fix for an underlying problem — if you're clipping constantly, your learning rate is probably too high or your model is poorly scaled, and the clipping is just masking it.
Threshold set to 5. On most steps the gradient's norm is around 2 — left untouched. But on one step it spikes to 80; clipping rescales it down to norm 5, keeping its direction. Without the clip, that single step might have sent the loss to infinity and crashed the training.
Keep the direction, cap the size — a guard against one ruinous step.
Clipping treats a symptom, not the disease. It reliably prevents blow-ups, but if it triggers on most steps, lower the learning rate or fix the architecture rather than relying on the clip.