Training Deep CNNs and Building Back Up

Why deeper isn't automatically better

By now you can build a working CNN: stack a few convolutional layers, pool here and there, flatten, and classify. A tempting next move is just more — twenty, fifty, a hundred layers. The intuition seems airtight: more layers means more capacity, so the network should fit the data at least as well. For most of the 2010s, this is exactly where deep learning got stuck. Naively stacking dozens of layers does not give you a better network. It gives you one that is harder to train — sometimes one you cannot train at all.

The first culprit is the gradient. Training works by backpropagation, which sends an error signal backward from the loss through every layer, multiplying by each layer's local derivative on the way. When you chain dozens of these multiplications together, the product tends to drift toward one of two disasters. If the per-layer factors are mostly smaller than one, the signal shrinks geometrically and arrives at the early layers as essentially zero — the vanishing gradient. If they are mostly larger than one, it blows up — the exploding gradient. Either way, the layers closest to the input stop receiving usable feedback, so they never learn the low-level features (edges, textures) that everything above them depends on.

The second culprit is subtler and was the real shock to the field: degradation. Researchers found that a 56-layer plain network had higher training error than an 18-layer one — not just worse test error, worse error on the very data it was being trained on. That is the key tell. A model that overfits memorises the training set, so its training error is tiny; here the deeper model could not even drive its training feature maps to a good fit. The extra layers were not overpowering the data, they were getting in the way of optimisation.

So the closing guide of this track is really about two stabilisers that, together, made depth finally pay off: residual connections, which give gradients a clear path home and make extra layers harmless by default, and batch normalisation, which keeps the numbers flowing through the network well-behaved. Once we can train deep, we will turn around and learn to build resolution back up — and finish with the modern CNN as a whole. The math lands in the next sections; here just hold the picture that depth is a training problem before it is anything else.

Residual blocks: learning the difference

Here is the fix that broke the depth barrier, and it is almost embarrassingly simple. Instead of asking a stack of layers to compute a brand-new output from its input, you ask it to compute only a change to its input, and then you add that change back on. The block's output is its input plus whatever the layers decided to adjust. The layers learn the residual — the difference between what came in and what should go out — rather than the whole thing from scratch.

Think of editing. Rewriting a paragraph from a blank page is hard; marking up an existing draft — "tighten this clause, cut that word" — is far easier, and if the draft is already perfect you simply write "no changes". A residual block works the same way: its default, do-nothing behaviour is cheap, and it only spends effort on the corrections that actually help.

Read this as: the block's output y equals the transformation 𝓕 applied to the input x, plus the original x added straight back. The symbols: x is the feature map coming into the block; 𝓕(x, {W_i}) is the residual function — a small stack of conv → batch-norm → ReLU → conv layers whose learnable weights are collected as {W_i}; and + x is the identity shortcut (or skip connection), a wire that carries the input forward unchanged. Crucially, 𝓕 no longer has to reproduce x — it only has to produce the correction. If the best thing a block can do is nothing, the layers just drive 𝓕 → 0 and y = x falls out for free. That is why adding more residual blocks never hurts: a stack of identities is always available as a fallback.

The second benefit is what saves the gradient. Differentiate the block with respect to its input and the addition splits cleanly into two terms.

The local derivative of the block is 1 (from the +x shortcut) plus ∂𝓕/∂x (from the layers). That constant 1 is the whole ballgame. In a plain network, if a layer's derivative is tiny — say 0.01 — then chaining 50 of them multiplies the gradient by 0.01⁵⁰, which is indistinguishable from zero: the signal dies. With the residual, each block contributes a factor near 1 + 0.01 = 1.01; across 50 blocks that is 1.01⁵⁰ ≈ 1.6 — the gradient arrives at the first layer essentially intact. The shortcut acts as a gradient highway: backprop can always flow straight down the +x wires, bypassing the throttling layers entirely. Vanishing gradients and degradation both fall away at once.

One practical wrinkle. The +x addition requires x and 𝓕(x) to have the same shape, but residual stages often change the channel count and spatial size (e.g. doubling channels while halving resolution). When the shortcut's shape doesn't match, you reshape it with a 1×1 convolution — the cheap channel-mixer from guide 4 — usually with a matching stride, so the skipped feature map lines up with 𝓕(x) before they are summed. It is a small adapter on the shortcut, not a change to the idea.

def residual_block(x, conv1, bn1, conv2, bn2, shortcut=None):
    # main path F(x): conv -> BN -> ReLU -> conv -> BN
    out = relu(bn1(conv1(x)))
    out = bn2(conv2(out))

    # identity shortcut; reshape with a 1x1 conv only if shapes differ
    identity = x if shortcut is None else shortcut(x)  # shortcut = 1x1 conv

    out = out + identity        # the +x that builds the gradient highway
    return relu(out)            # activation applied AFTER the addition

Batch normalization: keeping activations sane

Residual connections give gradients a path home, but there is a second source of training pain: as data flows forward, the scale of the activations inside the network drifts. Early in training the weights are random, so one layer might output values clustered around 0.1 and the next around 50; every layer is trying to learn while the statistics of its own input keep shifting under it. Batch normalisation tackles this directly by re-standardising the activations at each layer so they stay in a steady, predictable range.

The crucial detail for CNNs is what gets normalised together. Batch norm works per channel: for each of a convolutional layer's feature channels, it gathers all the activations of that channel across the whole mini-batch (and, for conv layers, across every spatial position too) and normalises that pool. A channel that has learned to detect, say, vertical edges gets its own mean and variance, independent of the colour-blob channel next to it. So a layer with 64 channels learns 64 separate normalisations.

Take the formula one symbol at a time. x is a single activation in the channel being normalised. μ_B is the mean of that channel's activations over the current mini-batch B, and σ_B² is their variance — both measured fresh for each batch. Subtracting μ_B re-centres the values on zero; dividing by the standard deviation √(σ_B²) rescales them to roughly unit spread. The ε is a tiny constant (like 10⁻⁵) added inside the square root purely so you never divide by zero when a channel happens to be constant. The result x̂ is the standardised activation: mean 0, variance ≈ 1.

Then comes the clever second half: y = γ x̂ + β. Here γ (scale) and β (shift) are learnable parameters, one pair per channel, trained by backprop like any weight. Why undo what we just did? Because forcing every activation to mean-0/variance-1 might be wrong for a particular channel — maybe ReLU works best when its input is shifted positive, or a feature is genuinely more useful with a wider spread. γ and β let the network choose its own scale and centre; in the extreme, setting γ = √(σ_B²) and β = μ_B recovers the original activation exactly. Normalisation becomes the helpful default, not a straitjacket, so no representational power is lost.

A concrete pass. Say one channel's activations across a mini-batch of four are [2, 4, 6, 8]. Then μ_B = 5 and σ_B² = ((−3)² + (−1)² + 1² + 3²)/4 = 5, so the std ≈ 2.24. The activation x = 8 becomes x̂ = (8 − 5)/2.24 ≈ 1.34. If this channel's learned parameters are γ = 1, β = 0, the output is just 1.34; if the network decided γ = 2, β = 0.5, it would emit 2(1.34) + 0.5 ≈ 3.18 instead. Same standardisation step, network-chosen final scale.

Why does this help so much? By pinning each layer's input distribution to a steady shape, batch norm stops every layer from chasing a moving target — its predecessors can reshuffle their weights without yanking the ground out from under it. Concretely you get faster, more stable training, tolerance for much higher learning rates (big steps no longer blow activations to infinity), and a mild regularising effect as a bonus: because μ_B and σ_B² are computed from a random mini-batch, each example is normalised by slightly different statistics every epoch, a gentle noise that discourages overfitting.

Transposed convolution: learnable upsampling

Everything so far flows one way: convolutions and pooling shrink a big image into a small, deep stack of feature maps — great for answering "what is in this picture?". But many tasks need the opposite. Semantic segmentation must label every pixel, so its output has to be the full image resolution. Image generation has to synthesise a large picture from a small code. For these we need to go back up — to turn a coarse, low-resolution feature map into a finer, higher-resolution one. The question is how to upsample in a way the network can learn, rather than a fixed rule like nearest-neighbour copying.

The answer is the transposed convolution — also called a fractionally-strided convolution, or loosely (and a bit misleadingly) a "deconvolution". It is best understood as running a convolution's data flow in reverse. A normal strided convolution takes a patch of input and collapses it to one output number; the transposed version does the mirror image — it takes one input number and spreads it across a whole kernel-sized patch of the output. Each input value is multiplied by the entire kernel, that scaled kernel is stamped onto the output grid, and where stamps from neighbouring inputs overlap, the contributions add up.

The name "transposed" is literal. Recall that a convolution can be written as multiplying the flattened input by a big sparse matrix of weights. The transposed convolution multiplies by the transpose of that same matrix — same weights, flipped role — which is exactly the operation that turns the small output shape back into the large input shape. Importantly, the kernel weights are learned by backprop, so the network discovers its own best way to fill in detail, rather than blurring with a fixed interpolation.

This is guide 2's output-size formula run backwards. Recall the forward convolution shrank the map by O_fwd = ⌊(W − K + 2P)/S⌋ + 1. The transposed version inverts it: O is the (larger) output size, W is the input size, K is the kernel size, P is the padding, and S is the stride — but each knob's job is flipped. The stride S no longer slides a window; it spaces the input values apart on the output grid, inserting S−1 gaps between them, which is precisely what magnifies the map (S = 2 roughly doubles the resolution). The padding P is now subtracted: instead of adding a border, it trims P pixels off each side of the result. So where the forward pass divided by S to shrink, the transposed pass multiplies by S to grow.

Let's verify it grows the map. Take a tiny 2×2 input (W = 2) with kernel K = 3, stride S = 2, padding P = 0: O = (2 − 1)·2 − 0 + 3 = 2 + 3 = 5, so a 2×2 map becomes 5×5. Sanity check the inverse direction: a forward conv on a 5×5 input with K = 3, S = 2, P = 0 gives ⌊(5 − 3)/2⌋ + 1 = 1 + 1 = 2 — back to 2×2. The two operations are mirror images, as promised. (With S = 1, P = 0, K = 3 the same 2×2 grows more modestly to (2−1)·1 + 3 = 4, i.e. 4×4.)

# Transposed conv as 'spread and sum' — the reverse of a strided conv.
# input X is HxW, kernel K is kxk, stride s. Output starts at zeros.
for i in range(H):
    for j in range(W):
        # stamp the whole kernel, scaled by this one input value,
        # at the output location set by the stride
        r, c = i * s, j * s
        output[r:r+k, c:c+k] += X[i, j] * K   # '+=' makes overlaps SUM
# (a final crop of P pixels per side applies the -2P term)

Data augmentation and regularization

Residual blocks and batch norm let us train deep CNNs. But training accuracy is not the goal — we want the network to generalise to images it has never seen. The single most effective tool here is also the most down-to-earth: data augmentation. Before each image goes into the network, you apply a random, label-preserving transformation — a horizontal flip, a random crop, a small rotation or scale change, a colour jitter (nudging brightness/contrast/hue), or harsher tricks like cutout (erasing a random patch) and mixup (blending two images and their labels). The cat is still a cat after you flip it, so the label stays valid while the pixels are fresh; the network effectively sees a much larger, more varied training set.

The deep insight is why this works, and it ties the whole track together. Augmentation is how you inject the invariances you want the model to have. If you want predictions to be unchanged when an object shifts, flips, or changes brightness, you show the network many shifted, flipped, and re-lit versions of each example and demand the same label every time. The model has no choice but to learn features that ignore those nuisances. You are, quite literally, teaching robustness by example.

This is also the honest follow-up to guide 3. Recall the caveat there: pooling gives a CNN only weak, approximate shift-invariance, and strictly the convolutions are translation-equivariant (shift the input, the feature map shifts with it) rather than fully invariant. The architecture alone does not guarantee that a face recognised in the centre is recognised in the corner, or that a tilted digit reads the same. Augmentation is how we make up the difference: invariances the network cannot prove for free, we drill in with examples. The two are complementary — architecture supplies a useful prior, augmentation supplies the robustness that prior cannot reach.

Two more regularisers round out the toolkit, attacking overfitting from the weight side rather than the data side. Weight decay adds a small penalty proportional to the size of the weights, nudging the network toward simpler, smaller-magnitude solutions that generalise better. Dropout randomly zeroes a fraction of activations during each training step, so no single unit can become indispensable and the network must learn redundant, robust representations; it is used less inside modern conv stacks (batch norm already regularises) but still common in the final dense layers. Reach for these when training accuracy is high but validation accuracy lags — the signature of overfitting from Section 1.

The modern CNN, and where vision goes next

Step back and the whole track snaps into one picture. Guide 1 showed a convolution is just a small window sliding over an image, computing weighted sums. Guide 2 made it a real layer — channels in and out, stride, padding — and gave us the output-size formula. Guide 3 added pooling, the growing receptive field, and the classic shrink-the-space-grow-the-channels pipeline that ends in a classifier. Guide 4 made convolutions efficient (1×1 mixers, depthwise-separable, dilated). This guide added the three things that make all of that go deep and reversible: residual blocks, batch normalisation, and transposed-convolution upsampling, plus the augmentation that makes it generalise. Each piece slots into the next.

Here is a modern backbone read end to end. A stem — one strided convolutional layer (ConvNeXt-style designs use a larger patch-like stem) — quickly downsamples the raw image and lifts it to many channels. Then come several residual stages; within a stage, blocks keep the resolution fixed, and between stages a strided block halves the spatial size while doubling the channels, so features grow more abstract and the receptive field widens. After the last stage, global average pooling averages each channel's map down to a single number, collapsing the spatial dimensions into one vector per channel. Finally a single linear layer maps that vector to class scores. Stem → residual stages → GAP → linear: that template, with batch norm inside every block, describes the overwhelming majority of CNN classifiers you will meet.

It is worth being honest about what makes a CNN a CNN — its inductive biases, the assumptions baked into the architecture. Locality: a convolution only looks at a small neighbourhood, betting that nearby pixels matter most. Weight sharing: the same kernel is reused at every position, so a feature detector learned in one place works everywhere, slashing parameters. Translation equivariance: shift the input and the feature maps shift in step. These priors are not free lunches — they are strong assumptions — but for images they are the right assumptions, which is why CNNs learn so much from so little data.

The honest comparison is with Vision Transformers (ViTs), which cut the image into patches and let global self-attention relate any patch to any other from layer one. ViTs throw away the CNN's built-in locality and translation priors, gaining flexibility — they can model long-range relationships directly — at a cost: with little data they tend to underperform CNNs, because they must learn from data the structure a CNN assumes for free. So with limited data, or limited compute, the CNN's priors still shine; with huge datasets, ViTs catch up and often surpass. Unsurprisingly the lines have blurred — hybrids abound, and modern convnets like ConvNeXt borrow Transformer training recipes while Transformers borrow convolutional stems. There is no single winner, only the right tool for the data budget.

Where to next. This track taught classification — one label per image. From here, object detection finds and boxes multiple objects, and semantic / instance segmentation labels every pixel (using the upsampling you just learned to climb back to full resolution). A dedicated Vision Transformers track picks up the attention thread. All of them stand on the same foundation you now own.

1. Stem: how is the image first downsampled and lifted to channels (stride? patch size?) — this sets the starting resolution and width.
2. Stages & blocks: how many stages, and is each block residual? Trace where channels double and resolution halves.
3. Normalisation & activation: batch norm (or a variant) and which non-linearity, and whether they sit before or after the addition.
4. Convolution flavour: plain, 1×1, depthwise-separable, or dilated? This tells you the cost/receptive-field trade-off (guide 4).
5. Receptive field: by the final stage, does one output unit's receptive field cover enough of the image for the task?
6. Head: GAP → linear for classification, or transposed-conv upsampling for dense prediction — this reveals what the network is built to output.