gradient clipping

Occasionally during training the gradient for a single step comes out enormous — a sudden cliff in the loss landscape, a rare hard example, or an unlucky interaction of weights — and a normal-sized learning rate times a giant gradient produces a giant step that flings the parameters far from anything sensible, often blowing the loss up to NaN and ruining the run. Gradient clipping is a simple seatbelt: before applying the update, cap how large the gradient is allowed to be. Most steps are unaffected, but the rare monster step is reined in to a safe magnitude, keeping training stable through the bumps.

The standard form is clipping by global norm. You compute the norm ‖g‖ of the entire gradient vector (the square root of the sum of squares of all gradient components across all parameters), and if it exceeds a chosen threshold c you rescale the whole gradient down to that norm: g ← g · min(1, c / ‖g‖). Crucially this preserves the gradient's direction — every component is scaled by the same factor — so you still step the right way, just no farther than length c. The simpler alternative, clipping by value, instead clamps each individual component into a range like [−c, c], which is cruder because it can change the gradient's direction, but is occasionally useful.

Gradient clipping is the standard defense against the exploding-gradient problem, the dual of vanishing gradients in which the backpropagated signal grows instead of shrinks. It is most associated with recurrent networks, where the same weights are applied over many time steps and small instabilities compound, but it is now routine in training transformers and large vision-language models, where a typical max-norm of around 1.0 is set as cheap insurance against the occasional loss spike that would otherwise derail a long, expensive run.

Two practical points. The clip must be applied after backpropagation computes the full gradient but before the optimizer step, and on the whole gradient (or per parameter group), not per layer in isolation, if you want global-norm semantics. And clipping is a stabilizer, not a cure for a fundamentally bad setup: if you are clipping aggressively on most steps, your learning rate is probably too high or your model is poorly conditioned, and you should fix the root cause rather than relying on the seatbelt every step.