residual connection

A residual connection is a deceptively simple wiring trick: instead of asking a stack of layers to compute a target transformation H(x) directly, you let the layers compute only F(x) and then add the original input back, producing the output H(x) = F(x) + x. The little arrow that carries x untouched around the layers and adds it at the end is the skip connection, and F(x) is the residual, the leftover that the layers must learn. The motivation is an empirical puzzle: before residual connections, making a plain network deeper past a certain point made it worse not just on test data but even on training data, which cannot be explained by overfitting. This degradation showed that very deep plain networks are simply hard to optimise.

Residual connections fix this by changing what the layers must learn. If the ideal mapping for some block is close to the identity (just pass the input through), a plain network must laboriously learn to reproduce its input through several nonlinear layers, which is hard; a residual block only needs to push F(x) toward zero, which is easy. More generally, the addition gives gradients a clean highway: during backpropagation the gradient of the loss flows directly through the identity term as well as through F, so even in a network hundreds of layers deep the early layers still receive a strong, non-vanishing gradient. This is the single change that made training networks of 100, 152, or even 1000 layers routine.

For the addition to work, F(x) and x must have the same shape; when a block changes the number of channels or the spatial resolution, the shortcut uses a 1x1 convolution (a projection shortcut) to match dimensions before adding. Residual connections are now nearly universal far beyond CNNs: every transformer block wraps its attention and feed-forward sub-layers in residual connections, and they are essential to the trainability of large language models, diffusion U-Nets, and almost all modern deep architectures. They are arguably the most important single architectural idea of the deep-learning era after convolution itself.