Sliding a Window: What a Convolution Really Does

Why we can't just flatten an image

Before we build anything, let's remember what an image actually is. From earlier tracks you already know it is a 2D grid of pixels — rows and columns of tiny squares — where each pixel stores a brightness number, say from 0 (black) to 255 (white). A colour photo simply stacks three such grids, one for red, one for green, one for blue; those stacked grids are the image's feature channels. So a modest 224×224 colour image is really three 224×224 tables of numbers: a little box of values that is 224 tall, 224 wide, and 3 deep.

The most obvious thing to try is the tool we already have from the basics of neural networks: a fully-connected (dense) layer, where every input wire connects to every neuron. But a dense layer expects a flat list of numbers, not a 3D box. So we would have to flatten the image — unroll every row, column and channel into one long vector. For our 224×224×3 image that vector has 224 × 224 × 3 ≈ 150,000 entries. And this is exactly where the naive plan falls apart.

The weight count for a single dense layer fed a flattened image.

Let's read that equation slowly. H and W are the image's height and width in pixels (224 each), and C is the number of colour channels (3 for RGB). Multiplying them, H × W × C, gives the length of the flattened input — about 150,000 numbers. N is how many neurons we put in the layer. Each neuron needs one weight for every input it listens to, so one neuron alone already carries ~150,000 weights. Stack a modest N = 1000 neurons and the single layer holds about 150 million weights — and that is for one layer, before we have even started stacking a deep network. The memory and compute simply explode.

The convolution operation, one window at a time

Here is the cure, and it is astonishingly simple. Instead of connecting a neuron to all 150,000 pixels, we give it a tiny window — a small grid of weights called a convolution kernel (often just 3×3). The convolution operation slides this window across the image. At every stop, it lays the kernel over the patch of pixels beneath it, multiplies each kernel weight by the pixel it covers, and adds up all those products into a single output number. Then it shifts one step and does it again. That sliding-multiply-add, repeated across the whole image, is the entire operation at the heart of every vision model.

The 2D cross-correlation: the output value at position (i, j).

Let's decode every symbol. I is the input image (a grid of pixel numbers) and K is the kernel (the little grid of weights). The pair (m, n) indexes positions inside the kernel — for a 3×3 kernel, m and n each run over 0, 1, 2 — so K(m, n) is one specific kernel weight. The pair (i, j) is the location in the output we are computing. The expression I(i+m, j+n) is the pixel sitting under kernel cell (m, n) when the window's corner is at (i, j). The double sum ∑ₘ∑ₙ just means 'add this up over the whole kernel window'. Read one term out loud: when m = 0, n = 0 we get K(0,0) · I(i, j) — the kernel's top-left weight times the pixel directly under it. Do that for all nine cells of a 3×3 kernel and add the nine products; that single total is S(i, j).

# 5x5 grayscale image patch I (one channel, values 0-9)
I = [[3, 1, 2, 0, 4],
     [1, 0, 5, 2, 1],
     [2, 3, 1, 4, 0],
     [0, 1, 2, 1, 3],
     [4, 2, 0, 3, 1]]

# 3x3 kernel K (a small blur-style kernel)
K = [[1, 2, 1],
     [2, 4, 2],
     [1, 2, 1]]

# --- Output position (0,0): K over rows 0-2, cols 0-2 ---
#   3*1 + 1*2 + 2*1
# + 1*2 + 0*4 + 5*2
# + 2*1 + 3*2 + 1*1
# = (3+2+2) + (2+0+10) + (2+6+1) = 28   -> S(0,0) = 28

# --- Slide one step RIGHT, position (0,1): cols 1-3 ---
#   1*1 + 2*2 + 0*1
# + 0*2 + 5*4 + 2*2
# + 3*1 + 1*2 + 4*1
# = (1+4+0) + (0+20+4) + (3+2+4) = 38   -> S(0,1) = 38

# --- Slide one more step RIGHT, position (0,2): cols 2-4 ---
#   2*1 + 0*2 + 4*1
# + 5*2 + 2*4 + 1*2
# + 1*1 + 4*2 + 0*1
# = (2+0+4) + (10+8+2) + (1+8+0) = 35   -> S(0,2) = 35

# First row of the output feature map: [28, 38, 35]
# (continue sliding down to fill the remaining rows)

One honest naming note, so the word never trips you up later. What we just computed — lay the kernel down as-is, multiply, add — is technically called cross-correlation. The textbook mathematical convolution is almost identical but first flips the kernel left-right and top-down before sliding. Why don't deep-learning people care? Because the kernel's numbers are learned, not chosen by hand: if a flip were needed, the network would simply learn the flipped weights. So the two are interchangeable in practice, and everyone calls the learned-weight version 'convolution' anyway. Just know the word cross-correlation so you recognise it when you read papers.

Kernels are feature detectors

We've seen how a kernel slides, but what does it actually do? Here is the most useful mental image: a convolution kernel is a tiny rubber stamp that 'lights up' wherever its own pattern appears in the image. Choose the numbers in the stamp cleverly and you can make it respond to vertical edges, or blurry regions, or sharp detail. The output it leaves behind is a map of where that pattern was found. Let's look at three classic hand-designed kernels to make this concrete.

(1) A vertical-edge (Sobel) kernel has positive weights down one side and negative weights down the other; its feature map glows bright along vertical boundaries — the left and right outlines of objects — and stays near zero across smooth areas. (2) A box-blur kernel is just nine equal weights of 1/9; sliding it replaces each pixel with the average of its neighbours, so its output is a softened, smeared copy of the input. (3) A sharpen kernel does the opposite: a large positive centre with small negative neighbours that exaggerates the difference between a pixel and its surroundings, making edges crisper. Same convolution operation every time — only the numbers in the kernel change.

The vertical-edge Sobel kernel.

Read this kernel column by column. The left column is all negative (−1, −2, −1), the centre column is zero (0, 0, 0), and the right column is all positive (1, 2, 1). So when we multiply-and-add, the kernel computes right-side brightness minus left-side brightness — exactly the left-to-right change in brightness. The middle row is doubled (−2 and 2) so the pixels right next to the centre count more, making the detector a touch sharper. Watch it work on a patch with a dark-to-bright edge, [[0,0,9],[0,0,9],[0,0,9]]: the response is (−1·0 + 0·0 + 1·9) + (−2·0 + 0·0 + 2·9) + (−1·0 + 0·0 + 1·9) = 9 + 18 + 9 = 36, a big number shouting 'edge here!'. Now feed it a flat patch where everything is 5: (−1·5 + 0 + 1·5) + (−2·5 + 0 + 2·5) + (−1·5 + 0 + 1·5) = 0 + 0 + 0 = 0 — the positives and negatives cancel, so a smooth region is correctly ignored. Remember: these exact numbers are what a network would otherwise learn on its own.

The feature map: the kernel's answer sheet

Let's pin down the term we've been using. A feature map is the complete 2D array of responses you get by sliding one kernel across the whole input — every S(i, j) value from the convolution operation, arranged back into a grid. One kernel in, one feature map out. If the kernel is an edge detector, its feature map is an 'edge report card' for the entire image.

The lovely part is that a feature map is itself just a grid of numbers, so you can look at it as a grayscale image. Read it like this: a bright pixel means 'the kernel's pattern was strongly present right here', and a dark pixel means 'that pattern is absent here'. So the edge detector's feature map is mostly dark, with bright streaks exactly where edges run. The feature map answers a single question — 'where does my pattern appear?' — for every location at once.

One kernel gives one feature map, but real layers run many kernels side by side — one tuned for vertical edges, one for horizontal edges, one for a blob of colour, and so on. Each produces its own feature map, and we stack them into a little box, exactly like the red/green/blue grids we started with. These stacked feature maps become the feature channels fed into the next layer — which is how the next guide builds a real convolutional layer and starts stacking depth. Channels in, richer channels out.

Weight sharing and translation equivariance

Let's collect the rewards, head-to-head against the dense-layer disaster of section 1. The first prize is weight sharing. Because the same small kernel scans the entire image, a convolutional layer holds only K × K weights per kernel — nine numbers for a 3×3 kernel — no matter whether the image is 224×224 or 4000×4000. Compare that with the 150 million weights a single dense layer demanded. And there's a bonus baked in: a feature detector that the convolution operation learns in one corner of the image automatically works everywhere else, because those same nine weights are applied at every position. We learn 'what an edge looks like' exactly once, not once per location.

The second prize has a longer name: translation equivariance. It means that if you shift the input — slide the cat ten pixels to the right — the feature map shifts by exactly the same amount, with its bright spots tracking the cat. Be careful to separate this from invariance. Equivariance means the output moves with the input; invariance means the output doesn't change at all when the input moves. The crisp analogy: your shadow is equivariant — step sideways and it slides exactly the same way; a thermometer on the wall is invariant — walk around the room and it reads the same temperature wherever you stand. Convolution gives us equivariance, not invariance.

Equivariance versus invariance, side by side.

Let's read the symbols. x is the input image. Tₜ is a translation — a shift by some amount t (say, ten pixels to the right). f is our convolution. The top line, f(Tₜ(x)) = Tₜ(f(x)), reads aloud as 'shift-then-convolve equals convolve-then-shift': you get the same result whether you slide the image first and then run the kernel, or run the kernel first and then slide the answer. That equality is exactly what 'the feature map moves with the input' means. The bottom line, f(Tₜ(x)) = f(x), is invariance: shifting the input would leave the output untouched. Convolution satisfies the top line, not the bottom — and feeling the difference between these two lines is the whole point of this section.

Why is convolution equivariant in the first place? Because the operation is identical at every location — the same kernel, the same multiply-and-add, applied at every stop. There is no special 'corner detector' that only exists in the top-left, so a pattern that moves simply triggers the same response at its new home. This is a genuinely good thing for early layers, which should report where features are. But sooner or later we want the opposite: a classifier should say 'cat' whether the cat is left or right — it wants invariance. The bridge between the two is pooling, a later operation that deliberately blurs out exact position, trading a little equivariance for invariance. That's the star of the third guide; here, just hold onto the clean fact that convolution moves its answers in lock-step with the image.