vanishing gradient
/ VAN-ish-ing GRAY-dee-ent /
The vanishing gradient is a problem that makes deep neural networks hard to train: the learning signal grows fainter and fainter as it travels back through the layers, until the earliest layers barely hear it at all. Picture a message whispered down a long line of people, each one repeating it a little more quietly — by the far end it's an inaudible mumble. The "message" here is the gradient, the signal that tells each weight which way to adjust, and the layers near the input are the people at the far end who can't make it out.
Where does the fading come from? Training adjusts weights by passing the error backward through the network, and at each layer the signal gets multiplied by a number. If those numbers are smaller than one — which happens easily with the flat tails of sigmoid and tanh activations — then multiplying many of them together drives the result toward zero, exponentially fast. Twenty layers of multiplying by, say, 0.3 leaves almost nothing. The early layers therefore receive a near-zero nudge and stop learning, while only the later layers improve. (The mirror-image trouble, where the numbers are bigger than one and the signal blows up, is called the exploding gradient.)
This problem stalled deep learning for years and explains a cluster of now-standard fixes. ReLU activations, which don't squash their positive side, keep the signal strong. Careful weight initialization keeps the early multipliers near one. Architectural tricks like residual connections give the signal a shortcut to skip layers, and batch normalization steadies it. None of these is a magic cure, but together they are why we can now train networks hundreds of layers deep — something that was practically impossible before they existed.
In a 20-layer network using sigmoid activations, each backward step might shrink the signal by a factor of about 0.25. After 20 layers that's 0.25 multiplied 20 times — roughly one part in a trillion. The first layer effectively gets told "adjust by nothing," so it never learns. Swap in ReLU and add residual shortcuts, and the signal survives the journey.
Multiplying many small numbers drives the gradient toward zero — early layers stop learning.
Vanishing and exploding gradients are two faces of the same coin — repeatedly multiplying by numbers below or above one. The standard defenses (ReLU, careful initialization, residual connections, normalization) tame it but don't abolish it.