vanishing gradient problem

Training a deep network means sending an error signal backward from the output through every layer, telling each weight how to change. The vanishing gradient problem is when that signal shrinks as it travels backward, so that by the time it reaches the early layers it has dwindled to almost nothing — and a weight that receives a near-zero gradient barely updates, so the first layers of a deep network learn agonizingly slowly or not at all. The network has depth on paper but cannot actually use it. The mirror image, the exploding gradient problem, is when the signal grows instead of shrinks, producing huge unstable updates and NaNs.

The cause is the chain rule, which makes backpropagation multiply many factors together. The gradient reaching an early layer is a long product of per-layer terms — roughly the layer weights times the derivatives of the activation functions, stage after stage. If those factors are typically less than one, their product decays exponentially with depth (0.8 multiplied by itself fifty times is essentially zero); if they are typically greater than one it explodes. Saturating activation functions made this acute in early networks: the sigmoid and tanh have derivatives at most 0.25 and 1 respectively, and they flatten to near-zero slope when their inputs are large, so deep stacks of them throttled the gradient to nothing. This is why training networks beyond a handful of layers was notoriously hard before the mid-2010s.

The fixes are largely architectural and together unlocked very deep modern vision. Non-saturating activations, above all the ReLU (which has derivative exactly 1 for positive inputs, so it does not shrink the gradient), replaced sigmoid and tanh. Careful weight initialization (He initialization for ReLU, Xavier/Glorot for tanh) sets the initial scale so that signal variance is preserved across layers rather than decaying. Normalization layers (batch and layer norm) keep activations in a healthy range so derivatives stay well-behaved. For recurrent networks, the gating of LSTMs and GRUs creates a near-identity path that carries gradient across many time steps.

The decisive fix for depth was the residual connection of ResNet (He et al., 2015). By adding a layer's input directly to its output — y = x + F(x) — a residual block creates a 'gradient highway': the +x term means the gradient can flow backward through the addition unchanged, bypassing the multiplicative decay of F entirely. This single idea made it routine to train networks of 50, 100, even 1000 layers, and it is the structural backbone of nearly every modern deep model, from ResNets to the transformer blocks inside ViT, DETR, and diffusion models.