The Residual Revolution: Training Networks 100 Layers Deep

The degradation problem: deeper got worse

By the time VGG and Inception arrived (the previous guide), the field had a tidy story: depth is good. Each extra layer composes simpler features into richer ones, so stacking more layers should let a network see more. Naturally, researchers tried going deeper still. And then something genuinely strange happened. Past a certain depth, adding layers did not just stop helping, it actively made the network worse, not only on test data but on the training data the network was directly optimising. A 56-layer plain stack reached a higher training error than a 20-layer one trained the same way.

Pause on how counterintuitive this is. Suppose your best 20-layer network is already good. Now build a 56-layer one and ask it to imitate the 20-layer network for its first 20 layers, then make the remaining 36 layers do nothing, just pass their input straight through (the identity mapping). A network that did this would exactly match the 20-layer net, never worse. So a deeper network can, in principle, always match a shallower one. There exists a setting of the weights that makes it at least as good. The fact that gradient descent could not find that setting, could not even learn 36 layers' worth of 'do nothing', tells us the problem is not the network's representational power. It is an optimisation problem: the right solution exists, but plain deep stacks are too hard to train toward it.

Why is 'do nothing' so hard to learn? This connects to the vanishing-gradient intuition you met in earlier training tracks. Training adjusts weights using gradients passed backward through every layer. In a long plain stack, that signal is repeatedly multiplied by each layer's local derivatives on the way back; if those factors are mostly smaller than one, the gradient shrinks geometrically and the early layers barely move. Even forcing many layers to reproduce their input exactly is a precise target that a flaky, attenuated learning signal struggles to hit. So depth created a training pathology, and the field needed a structural fix, not just a bigger GPU. That fix, the residual network, and its core building block, the residual connection, is the rest of this guide.

The residual connection: learn the correction

Here is the idea that broke the depth barrier, and it is almost embarrassingly simple. Normally we ask a block of layers to compute some desired output H(x) directly from its input x. The residual connection changes the question. Instead of forcing the block to produce the whole answer, we let it compute only the difference between the answer and the input, F(x) = H(x) − x, and then we add the input back at the end. The block no longer has to learn the full mapping; it only has to learn what to change about x.

Think of editing a draft. Rewriting an essay from a blank page is hard; marking up an existing draft with a few corrections is easy, and if the draft is already perfect, the easiest possible edit is to write 'no changes' and hand it back. A residual connection gives the network exactly that option. If the best thing a block can do is leave its input alone, it just needs to drive its learned part F toward zero, and the skip connection delivers the untouched input to the output. 'Do nothing' is no longer a delicate target buried under 36 layers of multiplication; it is the natural resting state of the block.

Trace the data flow in the figure carefully, because every later architecture reuses this shape. The input x enters and splits in two. On the main path it goes through a small stack, typically two 3×3 convolutions with a ReLU between them, producing the learned residual F(x). On the side path, the skip (or shortcut) carries x forward untouched. The two paths meet at an addition: we add F(x) and x element-by-element. The sum then passes through a final ReLU to become the block's output. Crucially, the addition requires F(x) and x to have the same shape (same height, width, and channel count) so they can be added position by position, a constraint we will revisit when shapes change.

The defining equation of a residual block.

Let us name every symbol. x is the block's input, the feature map (a stack of channels) arriving from the previous block. 𝓕(x, {Wᵢ}) is the learned residual: the function computed by the main path, which is a couple of convolution-plus-ReLU layers whose adjustable weights are written {Wᵢ} (the set of all weight matrices Wᵢ in those layers). The + x is the identity shortcut, literally adding the original input back in. y is the block's output that flows to the next block. Now the punchline of the whole guide, read straight off the equation: if learning pushes every weight in 𝓕 toward zero, then 𝓕(x,{Wᵢ}) = 0 and y = 0 + x = x. The block becomes the identity automatically. Concretely, imagine x is a single number, x = 3. A plain block would have to learn weights that output exactly 3 to preserve it. The residual block only needs 𝓕 = 0, giving y = 0 + 3 = 3 for free. Because identity is now the cheap default, stacking more residual blocks can never make training worse by construction, exactly the guarantee the plain deep stack lacked.

Why it works: gradient highways and batch norm

We have seen the forward-pass story (identity is cheap). The deeper reason residual nets actually train lives in the backward pass, where gradients flow. Picture a multi-lane road. In a plain deep network, every gradient must crawl back through every layer's local toll-booth, getting multiplied (and usually shrunk) at each one, the vanishing-gradient problem. The skip connection adds an express lane: a direct route that lets the gradient flow backward across the block without being filtered through its convolutions. Even when the convolutional lane is congested and barely passes a signal, the express lane keeps a strong gradient reaching the early layers.

The gradient flowing backward through one residual block.

This is the most important equation in the guide, so let us unpack each piece. L is the loss, the single number measuring how wrong the network currently is, which training tries to minimise. ∂L/∂x is what we want: how the loss changes if we nudge the block's input x, the gradient we must pass further back to earlier layers. ∂L/∂y is the upstream gradient, the signal that already arrived from the layers above. ∂𝓕/∂x is the residual path's own gradient, how the learned part F responds to changes in x. Apply the chain rule to y = 𝓕(x) + x and the derivative of that '+ x' term contributes a clean 1, while the 𝓕 term contributes ∂𝓕/∂x, giving the factor (1 + ∂𝓕/∂x). Here is why it matters: even if the convolutions are badly conditioned so that ∂𝓕/∂x shrinks toward 0, the bracket becomes (1 + 0) = 1, and the upstream gradient ∂L/∂y passes through essentially undiminished. The '+1' is the express lane written in calculus, a gradient path that cannot vanish. Numerically: if a 50-layer plain net multiplied factors of about 0.5 fifty times, the signal would fade to 0.5⁵⁰ ≈ 10⁻¹⁵ (gone). With residual blocks each factor is roughly (1 + small), so the product stays near 1 no matter how deep you go.

The express lane is half the engineering. The other half is batch normalization (batch norm), which ResNet leans on heavily, one BN layer after every convolution. Recall what it does: for each channel, it takes the activations across the current mini-batch, subtracts their mean and divides by their standard deviation so they have a tidy, roughly zero-mean, unit-variance distribution, then lets the network rescale and re-shift them with two learned parameters. The effect is to keep every layer's inputs well-conditioned, neither exploding to huge values nor collapsing to tiny ones as training proceeds. That stability is what lets you safely crank the learning rate and actually train 100+ layers. Residual connections and batch norm are partners: the skip keeps the gradient alive across depth, and batch norm keeps the activations on each path numerically sane so that gradient stays meaningful.

Inside ResNet: basic blocks, bottlenecks, and depth

Now the concrete architecture. ResNet ships as a family named by depth: ResNet-18, -34, -50, -101, and -152, where the number is roughly how many weight layers (convolutions plus the final fully-connected layer) are stacked. They all share the same skeleton, a single 7×7 convolution and pooling stem, then four stages of residual blocks, then global average pooling and a classifier, but they differ in how many blocks each stage holds and, crucially, which of two block designs they use.

The basic block is the one we drew in Section 2: two 3×3 convolutions (each followed by batch norm and ReLU) plus the skip. It is used in the shallower ResNet-18 and -34. But two 3×3 convs operating on, say, 256 channels are expensive: a 3×3 conv from 256 channels to 256 channels costs 3·3·256·256 ≈ 590k multiply-adds per output position. Stack that 50+ times and the bill explodes. So the deeper variants switch to the bottleneck block, which reuses the 1×1 trick from the previous guide.

The bottleneck has three convolutions: a 1×1 that reduces the channel count (say 256 → 64), a 3×3 that does the spatial work at the cheap reduced width (64 → 64), and a 1×1 that expands back up (64 → 256) so the skip-add still lines up. The name comes from the shape: wide, then pinched narrow in the middle, then wide again. Why this saves so much: the costly 3×3 now runs at 64 channels instead of 256, and the two 1×1 convs are cheap because a 1×1 has no spatial extent. Let us tally it. The reduce costs 1·1·256·64 ≈ 16k, the 3×3 costs 3·3·64·64 ≈ 37k, the expand costs 1·1·64·256 ≈ 16k, totalling about 70k multiply-adds per position, versus 590k for one 3×3 at full width. The bottleneck buys depth at a fraction of the compute, which is exactly how ResNet-50/101/152 stay affordable.

Two more pieces of plumbing let stages connect. First, downsampling: between stages the spatial resolution should halve (so later layers see a coarser, more abstract grid) while channels double. ResNet does this with a strided convolution, a conv that steps two pixels at a time instead of one, so a 56×56 map becomes 28×28 in a single layer rather than via a separate pooling step. Second, the projection shortcut. Remember the skip-add needs both paths to match in shape. At a stage boundary the main path changes both spatial size and channel count, so the identity x no longer matches F(x). The fix is to put a 1×1 convolution with the same stride on the skip path, just enough computation to reshape x (downsample it spatially and adjust its channels) so the addition lines up again. Inside a stage, where shapes are unchanged, the shortcut stays a plain, free identity.

# One ResNet bottleneck block (PyTorch-style pseudocode)
# in_ch=256 -> mid=64 -> out_ch=256, with an optional downsampling skip

class Bottleneck:
    def __init__(self, in_ch, mid, out_ch, stride=1):
        # 1x1 reduce: shrink channels (cheap) before the costly 3x3
        self.reduce = Conv1x1(in_ch, mid)
        self.bn1    = BatchNorm(mid)
        # 3x3 spatial conv at the narrow width; stride halves H,W when >1
        self.conv   = Conv3x3(mid, mid, stride=stride)
        self.bn2    = BatchNorm(mid)
        # 1x1 expand: restore channels so the skip-add lines up
        self.expand = Conv1x1(mid, out_ch)
        self.bn3    = BatchNorm(out_ch)
        # projection shortcut ONLY when shape changes (stride>1 or ch differ)
        self.proj = None
        if stride != 1 or in_ch != out_ch:
            self.proj = Conv1x1(in_ch, out_ch, stride=stride)

    def forward(self, x):
        identity = x if self.proj is None else self.proj(x)  # reshape skip if needed
        out = relu(self.bn1(self.reduce(x)))
        out = relu(self.bn2(self.conv(out)))
        out = self.bn3(self.expand(out))   # NOTE: no ReLU yet
        out = out + identity               # the residual addition
        return relu(out)                   # final activation AFTER the add

ResNeXt: cardinality, a third lever

Once ResNet worked, the obvious ways to make a network bigger were to go deeper (more blocks) or wider (more channels per layer). ResNeXt proposed a third, independent knob, and named it cardinality: the number of parallel transformation branches inside a block. Instead of one bottleneck doing 256 → 64 → 64 → 256, a ResNeXt block runs, say, 32 smaller branches side by side, each doing 256 → 4 → 4 → 256, and sums their outputs. Cardinality is simply how many such branches you have.

If this smells like the Inception module from the previous guide, you have the right instinct, both follow a split-transform-merge pattern: split the signal into parallel paths, transform each, then merge. But there is a sharp, deliberate difference. Inception's branches are all different (a 1×1 here, a 3×3 there, a 5×5 elsewhere), hand-designed and fiddly. ResNeXt's branches are all the same cheap topology, just repeated. That uniformity is the whole selling point: there is only one branch design to tune, and scaling the block means changing a single integer (the cardinality) rather than redesigning a zoo of differently-shaped paths. Simpler to reason about, simpler to scale.

The aggregated transformation at the heart of ResNeXt.

Let us name the symbols and see how this generalises the residual block. x is again the block input and y its output. C is the cardinality, the number of parallel branches. Each 𝓣ᵢ(x) is one transformation branch, in practice the same small bottleneck shape for every i, but with its own independent weights, so different branches learn different specialists. The Σ from i = 1 to C means we add up all C branch outputs element by element (merge), and the + x is our familiar residual shortcut. Sanity check: set C = 1 and a single branch, and the formula collapses to y = x + 𝓣₁(x), exactly the ordinary residual block from Section 2. ResNeXt is therefore a strict generalisation: turning the cardinality knob from 1 up to 32 adds parallel specialists without touching depth or the per-branch design.

The practical payoff: at a roughly fixed compute and parameter budget, increasing cardinality tended to raise accuracy more than spending those same resources on extra width or depth. Many parallel specialists beat one fat generalist. There is also a clean implementation detail that matters for the next guide. Running C identical branches that each touch only a slice of the channels is exactly a grouped convolution, a single conv layer split into C independent groups that do not mix channels across groups. So ResNeXt is, under the hood, a bottleneck whose middle 3×3 is a grouped conv with C groups. Hold onto grouped convolutions: the efficiency-focused architectures in guide 4, especially ShuffleNet, build directly on them to squeeze vision models onto phones.

DenseNet: connect everything to everything

ResNet asked: what if a layer only learns a correction to its input? DenseNet pushes the connectivity idea to its limit and asks a different question: what if every layer can see the outputs of all the layers before it? Inside a dense block, layer 1 feeds layers 2, 3, 4, …; layer 2 feeds layers 3, 4, …; and so on. Each layer receives, as its input, the stacked outputs of every preceding layer in the block. Where ResNet adds one express lane, DenseNet wires a dense mesh of shortcuts so information and gradients reach everywhere.

Dense connectivity: layer ℓ sees the concatenation of all earlier feature maps.

Symbol by symbol. x₀ is the input to the dense block, and x₁, …, x_{ℓ−1} are the feature maps produced by layers 1 through ℓ−1. The square brackets [x₀, x₁, …, x_{ℓ−1}] denote concatenation: line all those feature maps up side by side along the channel axis into one tall stack (this is the step that is not an addition). H_ℓ is layer ℓ's transformation, in DenseNet a composite of batch norm, then ReLU, then a convolution. x_ℓ is what layer ℓ outputs, which then itself joins the growing concatenation that later layers will read. Contrast this directly with ResNet's y = 𝓕(x) + x: there the earlier signal is summed into the new one and disappears as a separate entity; here it is stacked, preserved verbatim, so every later layer can still reach back and reuse the raw early features. That preservation, not blending, is the entire point.

Because each layer's output is preserved and reused, layers do not need to re-derive features that already exist, so each one can be deliberately thin, contributing only a small number of new channels. That number is DenseNet's headline hyperparameter, the growth rate k (often k = 12 or 32): every layer adds exactly k new feature maps to the stack. After ℓ layers the block holds x₀'s channels plus ℓ·k, growing linearly. Concretely, with a 64-channel input and k = 32, the inputs to successive layers are 64, then 96, then 128, then 160 channels, each layer adds its 32 and passes the whole pile on. Because k is small, the total parameter count stays surprisingly low, which is why DenseNets can match ResNet accuracy with markedly fewer parameters.

Tally the trade-offs. The wins: gradients flow superbly because every layer has a near-direct path to the loss through the concatenations (an even denser version of the gradient highway from Section 3); parameters are used efficiently thanks to the thin growth-rate layers; and feature reuse is maximal, since no information is overwritten. The cost: concatenation is memory-hungry. Adding feature maps (ResNet) leaves the channel count flat, but stacking them (DenseNet) makes activations pile up, and all those intermediate maps must be kept in memory for the backward pass, which is the practical reason DenseNets can be fiddlier to train on limited hardware. In practice DenseNet inserts transition layers (a 1×1 conv plus pooling) between dense blocks to compress the channel stack and halve the spatial size, keeping the growth in check. Two philosophies, then, from one revolution: ResNet adds a correction, DenseNet preserves and reuses, and almost every modern vision backbone is a descendant of one, the other, or both.