residual connection
/ rih-ZID-joo-ul kuh-NEK-shun /
A residual connection is a small wiring trick that made it possible to train truly deep networks — hundreds of layers — without them falling apart. The idea is almost embarrassingly simple: alongside the path where a layer transforms its input, add a shortcut that copies the original input straight through and adds it back to the layer's output. So instead of forcing each layer to reinvent everything from scratch, you only ask it to learn the residual — the small change, the leftover difference — on top of what already came in.
Picture editing a paragraph. You could rewrite it entirely at every pass, or you could keep the existing text and just mark up the edits. The second way is safer: if a particular pass has nothing useful to add, it can leave the text alone, and you have lost nothing. A residual connection gives a layer that same option. If a layer is not helping, it can learn to output near-zero and let the untouched input flow past via the shortcut — so adding more layers can no longer make things worse, only potentially better.
Why it mattered so much: before this, stacking many layers paradoxically hurt accuracy, because the training signal (the gradient) had to pass back through every layer and faded to nothing — the vanishing-gradient problem. The shortcut gives that signal a clear, direct path backward, so very deep networks finally trained well. Introduced in 2015 as ResNet, this one idea is now standard equipment in nearly every deep architecture, Transformers included. It is not free — it adds connections and memory — but the payoff in trainable depth was enormous.
Write a block's output as output = F(input) + input. The "+ input" is the residual connection. If the block learns F ≈ 0, the input simply passes through unchanged — the layer does no harm even when it has nothing to add.
Learn the change, not the whole thing — and let the original flow past for free.
"Residual connection" and "skip connection" are often used interchangeably. The key insight is that asking a layer to learn only the difference from its input is far easier than asking it to learn the full mapping — and it gives the training signal a shortcut home.