Deeper and Cheaper: VGG, the 1×1 Trick, and Inception

VGG: the beauty of doing one thing

In the last guide we watched AlexNet win by throwing a grab-bag of ideas at the problem: an 11×11 filter here, a 5×5 there, then some 3×3s, with strides and pooling chosen layer by layer. It worked, but it was a bit like a kitchen with a different knife for every vegetable. The team behind VGG asked a quieter question: what if we stop varying everything, pick the smallest sensible filter, and just repeat it? Their answer turned out to be one of the most influential design choices in computer vision.

VGG's philosophy is extreme uniformity. Almost every convolution in the network is the same: a 3×3 filter with stride 1 and padding 1. That padding-1 detail is what keeps the spatial size unchanged through a convolution, so a block of these layers neither shrinks nor grows the feature map — it only deepens the analysis. The only thing that changes the spatial size is a 2×2 max-pooling layer placed between blocks, which halves height and width. Compared with the irregular mix of big filters in AlexNet, VGG feels almost monastic: one move, repeated with discipline.

Here is the rhythm that makes VGG so easy to picture. Say you feed in a 224×224 image with 3 colour channels. Block 1 runs a couple of 3×3 convs producing 64 channels, then pools: now 112×112×64. Block 2 doubles to 128 channels, then pools: 56×56×128. Block 3 → 28×28×256, block 4 → 14×14×512, block 5 → 7×7×512. Notice the two simple rules working in lockstep: every pool halves the spatial size, and every block doubles the channels (until it caps at 512). Space shrinks, depth grows — the network trades 'where' for 'what' as you go up.

Why two 3×3 filters beat one 5×5

If small filters are so good, you might worry they are short-sighted — a 3×3 filter only ever looks at a tiny 3×3 patch. The trick that saves VGG is the receptive field: the region of the original input that a given neuron can ultimately 'see' once you account for all the layers beneath it. The surprise is that stacking small filters grows that window quickly, and a stack of 3×3s can see exactly as much as one big filter.

Let's build the window up by hand. One 3×3 conv: each output sees a 3×3 patch — obvious. Now stack a second 3×3 conv on top. Each neuron in this second layer looks at a 3×3 patch of the first layer's output. But each of those first-layer values already summarised a 3×3 patch of the input, and neighbouring ones overlap by shifting one pixel at a time. Lay them out and the combined view spans a 5×5 region of the original image. Add a third 3×3 layer and the window grows to 7×7. So two 3×3s = one 5×5, and three 3×3s = one 7×7, in terms of how far each neuron can see.

Seeing the same area is nice, but the real win is the price. Suppose a layer has C input channels and C output channels — a common situation deep inside a block. Compare the weight counts of one 5×5 conv against two stacked 3×3 convs:

Same receptive field, fewer weights: 18C² versus 25C².

Let's unpack every factor so nothing is magic. In one 5×5 conv, the term 5² = 25 is the kernel area (a 5-by-5 grid of weights), and C² = C·C is the channel pairing: every one of the C input channels connects to every one of the C output channels, so the kernel is replicated for each (input, output) pair. Multiply and you get 25C² weights. For the two-3×3 stack, 3² = 9 is the smaller kernel area, C² is the same channel pairing, and the leading 2 simply counts the two stacked layers. That gives 2·9·C² = 18C². Plug in a real number, say C = 64: the 5×5 needs 25·4096 ≈ 102k weights, while the two 3×3s need 18·4096 ≈ 74k — about 28% cheaper for the same field of view.

The 1×1 convolution: an MLP at every pixel

Now for an idea that baffles almost everyone the first time: the 1×1 convolution. Your instinct screams that a 1×1 filter is pointless — what can a single pixel-wide window possibly learn about an image? The resolution is to stop thinking about space. A feature map is not just height and width; at each location it has a whole stack of channels (64, 256, sometimes thousands). A 1×1 conv ignores spatial neighbours entirely and instead operates down through that stack of channels at one pixel.

Formally, at a single pixel a 1×1 convolution computes:

One small matrix–vector multiply, applied independently at every spatial position.

Let's name every symbol. At one pixel, x is the channel vector: the list of all C_in feature values stacked at that location (if the layer has 256 channels, x has 256 numbers). W is a weight matrix of shape C_out × C_in — that is, C_out rows, each row a set of C_in mixing weights. The output y is a new channel vector of length C_out: each entry of y is a weighted blend of all the input channels. The crucial part: the same matrix W is reused at every (x,y) position in the image. So a 1×1 conv is literally a tiny fully-connected layer — an MLP across channels — copy-pasted onto every pixel. It is pure channel mixing, zero spatial mixing.

Its killer application is dimensionality reduction: cheaply collapsing many channels into few before an expensive layer. This is the contribution of the Network-in-Network paper, and it sets up the bottlenecks we'll see in Inception. Concretely, suppose you have a 256-channel feature map and you want to feed a costly 3×3 conv. Going straight from 256→256 with a 3×3 costs 3²·256·256 ≈ 590k weights. Instead, first use a 1×1 conv to squeeze 256→64 (cost 1²·256·64 ≈ 16k), then run the 3×3 at 64→64 (cost 3²·64·64 ≈ 37k). Total ≈ 53k weights — more than 10× cheaper, and the 1×1 learns which 64 directions to keep.

import torch.nn as nn

# A 1x1 convolution IS a per-pixel linear layer across channels.
# kernel_size=1 means it never looks at spatial neighbours.
reduce = nn.Conv2d(in_channels=256, out_channels=64, kernel_size=1)

# Input: batch of 8 feature maps, 256 channels, 28x28 spatially.
x = torch.randn(8, 256, 28, 28)
y = reduce(x)          # -> shape (8, 64, 28, 28): channels squeezed, H/W untouched

# Parameter count: C_out * C_in * 1 * 1  (+ C_out biases)
#   = 64 * 256 * 1 * 1 + 64  = 16,448 weights -- tiny.
print(y.shape, sum(p.numel() for p in reduce.parameters()))

Global average pooling: dropping the giant head

Remember the villain from guide 1's parameter count? In AlexNet and VGG, the convolutional part is comparatively lean, but the fully-connected classifier head at the end is a monster — VGG's first FC layer alone holds over 100 million weights, the bulk of the whole network. The Network-in-Network paper proposed a startlingly simple replacement: global average pooling.

The idea: instead of flattening the final feature map into a giant vector and running huge FC layers, just average each final feature map down to a single number. If the last conv block outputs, say, a 7×7×512 tensor, you collapse each 7×7 map to its average, giving exactly 512 numbers — one value per channel. Those 512 numbers feed straight into the final classifier. No flattening, no hundred-million-weight layer.

Here is the intuition that makes it click. By this depth, each of the 512 feature maps has learned to respond to some high-level pattern — think of one as a 'wheel detector', another as a 'fur detector'. Averaging a map asks: *across the whole image, how strongly did this detector fire?* That single average is the detector's confidence vote. Feeding those votes to the classifier is wonderfully direct: each class is essentially a weighted tally of which detectors lit up.

The Inception module: why choose one filter size?

VGG committed to 3×3 everywhere. The Inception module takes the opposite stance and refuses to commit at all. Its bold question: in any given layer, why must we pick one filter size? A cat's whisker is a fine detail; the cat's body is a coarse one. Maybe the network shouldn't be forced to choose. So Inception runs several filters in parallel on the same input — a 1×1, a 3×3, a 5×5, and a pooling branch — and lets the data decide what is useful at each spot.

Think of it as consulting a panel of specialists at once: one expert squints at fine texture (small filter), one steps back for medium structure (3×3), one takes in the broad context (5×5), and one summarises the surroundings (pooling). Each branch produces its own feature maps. The module then concatenates these outputs — and this is the part beginners must get right: the branches are stacked along the channel dimension, not added together. If the branches output 64, 128, 32, and 32 channels, the module's output has 64+128+32+32 = 256 channels, all sharing the same height and width. The network keeps everyone's opinion side by side and lets later layers weigh them.

There is an obvious problem: running a 5×5 conv on a fat 256-channel input is brutally expensive, and concatenation only makes the channel count balloon further. This is where the previous section pays off. Inception inserts 1×1 convolutions as bottlenecks right before the costly 3×3 and 5×5 branches, using exactly the Network-in-Network channel-reduction trick. The 1×1 squeezes, say, 256 channels down to 64, the expensive conv runs on that slim input, and the cost collapses. The 1×1 bottleneck is the quiet hero that makes the whole parallel-branch idea affordable.

GoogLeNet: deep, wide, and surprisingly cheap

Now stack those modules. GoogLeNet is essentially nine Inception modules piled on top of one another (with a couple of plain convs and pools at the very start), giving a network 22 layers deep. Going deeper usually means a parameter explosion — yet GoogLeNet has only about 5 million parameters. For contrast, VGG-16 weighs in at around 138 million. That is the efficiency leap: roughly a fraction of VGG's weights, from a network that is actually deeper.

Where did all those weights go? Two savings we built up across this guide. First, the 1×1 bottlenecks inside every Inception module keep the expensive 3×3 and 5×5 branches running on slimmed-down channels. Second, GoogLeNet ends in global average pooling instead of VGG's enormous fully-connected head — deleting the single heaviest component. Deep and wide on the inside, lean at the ends.

One more detail worth meeting now because it foreshadows the next guide: auxiliary classifiers. During training, GoogLeNet sprouts two small extra classifier heads partway up the network, each making its own prediction and contributing to the loss. Why? In a 22-layer net, the training signal (the gradient) computed at the very end has to travel a long way back to teach the early layers, and it tends to fade on the journey. The auxiliary heads inject fresh gradient directly into the middle of the network, giving early layers a clearer 'study guide'. They are removed once training finishes.

Put the two designs side by side. VGG-16: ~138M parameters, beautifully simple, a workhorse backbone — but heavy. GoogLeNet: ~5M parameters, 22 layers, comparable or better top-5 accuracy on ImageNet (both around the high-80s/low-90s percent), at a fraction of the memory and compute. The lesson of this guide is that cleverness in architecture — small stacked filters, 1×1 channel mixing, parallel branches, and a pooled head — buys you depth and accuracy far more cheaply than brute force.