Pooling, Receptive Fields, and a Whole CNN

Pooling: summarizing a neighborhood

In the last guide you built a convolutional layer: a stack of filters that slides over an image and produces a feature map for each filter, where a high value means 'my pattern showed up here.' A pooling layer is the simple, quiet partner that usually comes right after. Its only job is to shrink each feature map by summarising every small window of it into a single number. Slide a tiny window (typically 2x2) across the map, replace the whole window with one summary value, and move on. There are no weights to learn here at all — pooling follows a fixed rule, like 'take the biggest value' or 'take the average.' That is why we call it a parameter-free downsampler.

Why bother summarising at all? Three reasons, and they reinforce each other. First, shrinking the maps saves memory and compute: halving height and width turns a 32x32 map into a 16x16 map, which is four times fewer numbers for every later layer to process. Second, it buys robustness to tiny shifts — if the edge or texture that lit up a cell moves by one pixel, the summary of its neighborhood often stays the same, so the network stops caring about exact pixel positions. Third, it lets later layers see a larger area of the original image with the same small kernel, because each pooled cell already stands in for a 2x2 patch of what came before. We will make that third point precise when we reach receptive fields.

Max vs average pooling

There are two classic rules, and the cleanest way to feel the difference is to run both on the exact same window. Take one 2x2 patch of a single feature map holding the values 1, 7, 3, 2. Max pooling keeps the single largest value in the window; average pooling keeps the mean of all the values. Same patch, two different summaries — and the choice changes what kind of information survives.

# Pooling is applied per-channel; this is ONE 2x2 window of ONE feature map.
window = [[1, 7],
          [3, 2]]

# Max pooling: keep the single strongest activation in the window.
max_pool = max(1, 7, 3, 2)          # -> 7

# Average pooling: take the mean of all four values.
avg_pool = (1 + 7 + 3 + 2) / 4      # -> 3.25

Read the two numbers as answers to two different questions. Max pooling's 7 answers 'did this feature fire ANYWHERE in this region?' — it grabs the strongest piece of evidence and throws away the rest, so a single strong response survives even if its three neighbors are weak. Average pooling's 3.25 answers 'how strong is this feature across the region on average?' — it preserves overall intensity and is smoother, because every value, strong or weak, gets a vote. Notice max is pulled up by the lone 7, while the average is dragged down by the three small numbers around it.

Practical guidance: max pooling dominates inside the pooling layers of classification backbones, because for recognising an object you mostly care whether a telltale feature appeared at all, not its average brightness. Average pooling shows up where a smooth, holistic summary matters — most famously as global average pooling at the very end of a network, which is the next section. One thing never changes between the two: pooling is always applied independently to each channel. A 16x16x64 volume pooled with a 2x2 window becomes 8x8x64 — the 64 channels are summarised separately and the channel count is untouched.

Global average pooling: from maps to a vector

At the end of a CNN you have a stack of feature maps, but a classifier needs a flat list of numbers it can weigh into class scores. Global average pooling (GAP) is the modern bridge. It is average pooling taken to its extreme: instead of a 2x2 window, the window is the ENTIRE feature map. So GAP collapses each full map to a single number, turning an H x W x C volume into a length-C vector. One number per channel, nothing left of the spatial layout.

Unpacking it: A_c is the c-th feature map (one full 2D grid of activations), and A_c(i,j) is the value at spatial position row i, column j of that map. The double sum adds up every value in the map, and H·W is the number of positions you summed over — the total number of cells, so dividing by it turns the sum into a plain average. The result z_c is one scalar: the single number that channel c contributes to the output vector. Do it for all C channels and you have your length-C vector. Concretely, if a 7x7 map has 49 activations summing to 98, then z_c = 98 / 49 = 2.0. The geometry is gone; what survives is 'on average, how strongly did this channel's feature appear anywhere in the image?'

Why this replaced the old head. The classic ending was 'flatten the whole volume into one long vector, then feed it to a big dense layer.' Flattening a 7x7x512 volume gives 25,088 numbers, and connecting that to even a modest layer costs millions of weights — a huge parameter count that loves to overfit. GAP has zero parameters, so it slashes the parameter budget and the overfitting that comes with it. It also accepts any input size: flatten demands a fixed H and W (more pixels means a longer vector and a broken dense layer), but GAP averages whatever grid it is handed and always yields exactly C numbers. Finally it is interpretable — because each channel becomes one direct number, you can read each feature channel as a piece of evidence the final layer weighs for or against each class.

Trace the standard head once: suppose the last convolutional block outputs a 7x7x512 volume. GAP averages each of the 512 maps (49 cells each) into one number, producing a 512-vector. A single linear layer maps those 512 numbers to, say, 10 class scores, and softmax turns the scores into probabilities that sum to 1. That tidy GAP -> linear -> softmax chain is the entire classifier head — and it is exactly the ending we will bolt onto the full pipeline in the last section.

Receptive field: how much each neuron sees

Here is a question that decides what a network can possibly recognise: when one unit deep inside the network produces a number, how much of the ORIGINAL input image was allowed to influence it? That region is its receptive field. A unit in the first layer might only see a 3x3 patch of pixels; a unit ten layers deep might effectively see most of the image. A neuron can never detect a whole cat if its receptive field only covers a whisker — so growing the receptive field is how a network graduates from seeing texture to seeing objects.

The central insight is about pace. Stacking small convolution kernels grows the receptive field gradually — each extra 3x3 layer adds a thin ring of context. But pooling and stride grow it rapidly, because after you downsample, every later step covers twice the original ground per cell. So two knobs control how fast the view widens: how many small convs you stack (slow, fine growth) and how much you pool or stride (fast, coarse growth).

Reading the recurrence: RF_l is the receptive-field size after layer l, and RF_{l-1} is what it was after the previous layer. K_l is this layer's kernel size, so (K_l − 1) is how many extra input cells this layer reaches beyond a 1x1 pass. The product of all previous strides, S_1 through S_{l−1}, is the key multiplier: each earlier pooling or strided layer (stride > 1) inflates this factor, which is exactly why a downsampling layer makes every LATER layer's receptive field grow faster. Start from RF_0 = 1 (a single input pixel) with an empty stride product of 1. Walk two stacked stride-1 3x3 convs: layer 1 gives RF_1 = 1 + (3−1)·1 = 3; layer 2 gives RF_2 = 3 + (3−1)·1 = 5. There is the 5x5 from the figure. Add a third identical conv and you get 5 + 2·1 = 7 — the famous result that two stacked 3x3 convs match one 5x5 kernel, and three match one 7x7, while using fewer weights. That is why deep stacks of cheap 3x3 convs replaced single big kernels.

Equivariance, invariance, and what pooling buys

Recall translation equivariance from guide 1: because convolution uses weight sharing — the same filter applied at every location — if the input shifts, the feature map shifts the same way. Move the cat right by ten pixels and its 'ear detector' response moves right by ten pixels too. That is equivariance: the output tracks the input's position. It is the right behavior for a layer that detects parts, but a classifier ultimately wants something stronger — it should answer 'cat' regardless of where the cat sits. That stronger property is invariance: the output should not change when the input shifts.

Pooling is what nudges the network from equivariance toward invariance. Picture a max pooling window where the strongest activation sits in the top-left cell. Shift the feature inside that window by one pixel so it lands in the top-right cell instead: the maximum is the same value, so the pooled output is unchanged. The layer has become locally insensitive to exactly where, within the window, the feature fired — which is precisely the small-shift robustness an object classifier wants. Stack several pooling stages and these little invariances compound; finish with global average pooling, which averages over the whole map, and position information is discarded almost entirely, leaving a near position-agnostic summary per channel.

Stacking it all: a complete CNN pipeline

Now assemble the pieces into the canonical architecture. The repeating unit is a block: [ convolutional layer -> ReLU -> (optionally another conv -> ReLU) -> pool ]. Stack a few of these blocks, and as you go deeper you widen the number of feature channels while pooling shrinks the height and width. When the spatial grid is small, finish with the head from section 3: global average pooling -> linear -> softmax. That single pattern, repeated and then capped with a GAP head, is the backbone of an enormous family of image classifiers.

The shape of the whole thing is a pyramid: spatial resolution falls while semantic depth rises. Early on the maps are large and shallow (many pixels, few channels, each channel a simple edge-like feature); deep down they are small and deep (few pixels, many channels, each channel a rich abstract concept). And one ingredient makes the stacking worthwhile — the ReLU nonlinearity from the earlier AI tracks, which zeroes out negatives after each conv. Without it, stacking convs would collapse into a single linear operation; with it, the network can learn genuinely non-linear patterns. Each feature map you carry forward is the output of conv-then-ReLU, never raw conv.

# A small CIFAR-style classifier. Shapes shown as (H, W, C).
x = image                      # (32, 32, 3)

# Block 1: two 3x3 convs (pad='same'), then halve H and W.
x = relu(conv3x3(x, 32))       # (32, 32, 32)
x = relu(conv3x3(x, 32))       # (32, 32, 32)
x = maxpool2x2(x)              # (16, 16, 32)

# Block 2: widen channels to 64.
x = relu(conv3x3(x, 64))       # (16, 16, 64)
x = relu(conv3x3(x, 64))       # (16, 16, 64)
x = maxpool2x2(x)              # ( 8,  8, 64)

# Block 3: widen channels to 128.
x = relu(conv3x3(x, 128))      # ( 8,  8, 128)
x = relu(conv3x3(x, 128))      # ( 8,  8, 128)
x = maxpool2x2(x)              # ( 4,  4, 128)

# Head: collapse space, then classify into 10 classes.
x      = global_average_pool(x)  # (128,)  one number per channel
logits = linear(x, 10)           # (10,)
probs  = softmax(logits)         # (10,)  class probabilities (sum to 1)

Step back and you have built a complete image-classification CNN: convolutions detect features, ReLU adds nonlinearity, pooling shrinks the maps and buys shift-robustness while receptive fields quietly grow until deep units see whole objects, and a GAP head turns the final maps into class probabilities. Two honest gaps remain. This network is correct but expensive — guide 4, 'Smarter, Lighter Convolutions,' will make the convs far cheaper without losing accuracy. And we have only described the forward pass — guide 5, 'Training Deep CNNs and Building Back Up,' will show how to actually learn all these filter weights, and how to stack the architecture much deeper without it falling apart.