The Sliding Window: Spatial Filtering and Smoothing

Beyond single pixels: why neighbourhoods matter

In the last guide, every operation we did treated each pixel as an island. To brighten an image we added a number to every pixel independently; to threshold it we asked, one pixel at a time, "is this value above 128?" These are called point operations because the new value of a pixel depends only on the old value of that same pixel — never on its neighbours. That is powerful, but it has a hard ceiling: a single number simply cannot tell you whether you are looking at the smooth middle of a wall, the crisp boundary of a doorway, or a speck of camera grain.

Here is the key intuition. Imagine I show you one handwritten letter, completely on its own, and it could be either a sloppy "c" or a sloppy "e". You genuinely cannot decide. But show you the same shape inside the word "c_t" versus "m_t" and your brain instantly fills it in. The letter only becomes meaningful in the context of the letters around it. Pixels are exactly the same: a single grey value of, say, 130 is meaningless, but if its neighbours are all around 130 you are in a smooth region, and if they jump from 40 to 220 right beside it, you are sitting on an edge.

So we make a leap to neighbourhood operations. Instead of looking at one pixel, we look at a small square patch of pixels centred on the one we care about — its neighbourhood, also called a window. A 3×3 window is the pixel plus its eight immediate neighbours; a 5×5 window reaches one ring further out. The rule we are building toward is simple to state: to compute one pixel of the output image, look at the window of pixels around the matching location in the input image, combine those numbers in some chosen way, and write the single result into the output. Do that for every location and you have filtered the whole image.

Almost every classical image filter — blurring, sharpening, edge detection, embossing — is just this one pattern with a different recipe for "combine the window." The whole machine is called spatial filtering, and the rest of this guide builds it up carefully: first the sliding-window mechanism itself, then what to do at the image border, and finally two concrete filters (the mean blur and the Gaussian blur) that turn the machinery into something visibly useful.

Convolution: the sliding window at the heart of filtering

Now the central idea of this entire track. The recipe for "combine the window" is itself a small grid of numbers called a kernel (also a filter or mask). A 3×3 kernel is just nine weights. We lay this kernel on top of the image so it covers a 3×3 patch, multiply each kernel weight by the pixel value sitting underneath it, add up all nine products, and that single sum becomes the new value of the centre pixel. Then we slide the kernel one step to the right and repeat, marching across the whole image like reading a page. This sliding, multiply-and-add procedure is called convolution.

A useful mental picture: the kernel is a little stencil that computes a weighted average. Think of a transparent stamp with a number written in each of its nine cells. Wherever you press it down, it reads the nine pixels showing through, weights each one by its printed number, and prints back a single blended value. Different stamps make different effects — a stamp of all equal weights blurs, a stamp with a big positive centre and negative edges sharpens — but the action of stamping is always the same: overlap, multiply, add.

Spatial filtering written out: output = sum over the window of weight times pixel.

Let us decode every symbol, because this one formula runs the whole track. f is the input image, so f(x+s, y+t) is the brightness of the input pixel at row y+t, column x+s. w is the kernel, the grid of weights, indexed by the small offsets s and t; for a 3×3 kernel both s and t run over −1, 0, +1, so (s,t)=(0,0) is the centre weight and (s,t)=(−1,−1) is the top-left one. g is the filtered output, and g(x,y) is the one number we are computing for output location (x,y). The double sum just says: step over every cell of the kernel window, multiply that weight by the input pixel it currently covers, and add all the products together. In three words: overlap, multiply, add.

Time for one fully worked 3×3 example by hand — do not skip this; it is the moment convolution clicks. Suppose the input patch under the kernel is, reading row by row, [10, 10, 10 / 10, 50, 10 / 10, 10, 10] — a flat grey region with one bright spike of 50 in the middle. Take a simple averaging kernel where every weight is 1/9. Each product is (1/9)×pixel, and the sum is (1/9)×(10+10+10+10+50+10+10+10+10) = (1/9)×130 ≈ 14.4. So the bright 50 gets pulled down to about 14 — the spike was averaged away by its calm neighbours. That single output number, 14, is written to the centre location of g, and then the kernel slides on to the next position.

Edges of the image: padding and stride

There is a practical snag we glossed over. When the kernel sits on a pixel in the very corner of the image, part of it hangs off the edge into nothing — for a 3×3 kernel centred on the top-left pixel, five of its nine cells have no pixel underneath. We cannot multiply a weight by a value that does not exist, so we need a policy for the border. This is the problem of padding.

The analogy: imagine hanging wallpaper, but the pattern you are stamping needs to reach slightly past the edge of the wall. Padding is how you decide what to pretend is there in that overhang. Three common choices: zero-padding assumes everything off-edge is black (value 0) — simplest, but it can darken and create a faint dark rim near borders. Replicate (edge) padding copies the nearest real edge pixel outward, like smearing the border colour off the wall — usually the most natural for blurring. Reflect padding mirrors the image across its edge, as if a wall met a mirror, which avoids the artificial flat band that replicate can produce. None is universally correct; they are just different reasonable guesses about the unknown.

The companion idea is stride: how far the kernel jumps between stops. With stride 1 (the default for almost every smoothing or edge filter) the window moves one pixel at a time and the output has essentially one value per input pixel. With stride 2 it skips every other position, sampling the image more coarsely and producing a smaller output. Most of this track lives at stride 1, but stride is worth meeting now because it becomes a load-bearing idea in the later CNN tracks, where strided convolutions are a standard way to shrink feature maps.

Output width as a function of input width, kernel size, padding, and stride.

This little formula is pure size bookkeeping, and every symbol is concrete. W is the input width in pixels; k is the kernel size (3 for a 3×3 kernel); p is how many padding pixels you add on each side; s is the stride; out is the resulting output width. The brackets ⌊ ⌋ mean floor — round down to the nearest whole number, since you cannot have a fractional pixel. The intuition reads left to right: W − k is how far the kernel's centre can travel before it runs off the edge (you lose a margin equal to the kernel), +2p adds back the room the padding buys you (p on the left and p on the right), dividing by s accounts for taking bigger hops, and the +1 counts the very first position where the kernel starts.

Plug in real numbers so it stops being abstract. Take a W = 100 wide image, a k = 3 kernel, no padding (p = 0), stride s = 1: out = ⌊(100 − 3 + 0)/1⌋ + 1 = ⌊97⌋ + 1 = 98. So with no padding the image quietly shrinks from 100 to 98 — you lose one pixel of border on each side. Now add p = 1 of padding: out = ⌊(100 − 3 + 2)/1⌋ + 1 = ⌊99⌋ + 1 = 100, exactly the original size. That is the standard trick for keeping an image the same size through a filter: pad by p = (k−1)/2, which for a 3×3 kernel is 1. Finally try stride s = 2 with that same padding: out = ⌊99/2⌋ + 1 = ⌊49.5⌋ + 1 = 49 + 1 = 50 — roughly half size, exactly as the bigger hops suggest.

The mean filter: the simplest blur

Now we finally fill the kernel with real numbers and watch the machine do something. The most intuitive smoothing recipe is: replace each pixel by the plain average of its neighbourhood. The kernel that does this is the box or mean filter — for a 3×3 window, every one of the nine weights is simply 1/9.

import numpy as np

# A 3x3 mean (box) kernel: nine equal weights of 1/9.
kernel = np.ones((3, 3), dtype=np.float32) / 9.0
#  [[1/9, 1/9, 1/9],
#   [1/9, 1/9, 1/9],
#   [1/9, 1/9, 1/9]]

def mean_filter_pixel(patch):
    # patch is the 3x3 window of input pixels under the kernel.
    # 'Overlap, multiply, add' == element-wise product then sum.
    return float(np.sum(patch * kernel))   # identical to patch.mean()

Every weight equals 1/N, so the filter is a plain average over the window.

Reading the formula: this is exactly the general convolution from Section 2, but with the weight w(s,t) pinned to the constant 1/N for every cell. N is the number of pixels in the window — for a 3×3 it is k·k = 3·3 = 9, for a 5×5 it is 25. Because every neighbour is multiplied by the same 1/N and the products are summed, the result is literally the arithmetic mean of the window. Picture it as a democratic vote: every neighbour gets one equal ballot, and the output is the consensus. That equality is the filter's strength and its weakness at the same time.

Why does averaging reduce noise? Noise is random — at a given pixel the sensor might have read a little too high, while its neighbour read a little too low, with no pattern to it. When you average nine such pixels, the random over-shoots and under-shoots tend to cancel out, while the true underlying brightness (which is roughly the same across all nine) survives. Recall our worked example: the lone spike of 50 in a sea of 10s collapsed to about 14. If that 50 was a noise glitch, the mean filter just cleaned it up beautifully.

But why does the same filter blur? Because it has no idea what structure it is sitting on. At a real edge — say nine pixels reading [10, 10, 10, 200, 200, 200, 200, 200, 200], a clean step from dark to light — the mean filter still just averages them to about 137, smearing the crisp boundary into a soft grey ramp. The democratic vote treats an edge pixel exactly like a flat-region pixel: it cannot tell a meaningful jump from a meaningless wobble, so it waters both down equally. That is the fundamental tension of smoothing, and it is precisely the flaw the next filter, and then the next guide, set out to fix.

Gaussian blur: smoothing that respects distance

The mean filter's flaw is that it gives a far-away corner of the window the same vote as the centre pixel itself. But intuitively, a pixel right next to you should tell you more about your true value than a pixel three steps away. The fix is to make the weights fall off with distance — nearer neighbours count more, distant ones count less. The classic way to do this is the gaussian blur, whose kernel is a bell-shaped hill of weights: tall in the centre, tapering smoothly toward the edges.

The 2D Gaussian: a smooth hill of weights centred on the pixel.

Let us unpack it carefully. x and y are the offset of a kernel cell from the centre — the centre cell is (0,0), the one directly right is (1,0), and so on, so x²+y² is just that cell's squared distance from the middle. σ (sigma) is the standard deviation, the single knob that controls how wide the hill is — its "reach" or blur radius. The exp(−distance²/2σ²) term is the heart of it: at the centre the exponent is 0 and exp(0)=1 (maximum weight), and as you move outward the weight shrinks fast, because distance enters squared. The front factor 1/(2πσ²) is a normaliser chosen so all the weights add up to 1 — that keeps the image's overall brightness unchanged (no accidental darkening or brightening).

σ is the dial you actually turn. A small σ makes a tight, narrow hill: weight is concentrated almost entirely on the centre pixel, so very little blurring happens and fine detail survives. A large σ spreads the hill wide: many neighbours get meaningful weight, so the image is smoothed strongly and softly. Two more properties to notice. First, because the formula depends only on x²+y² (distance, not direction), the kernel is rotationally symmetric — it blurs equally in every direction, so a Gaussian blur has no preferred orientation and introduces no streaky artefacts. Second, it is perfectly symmetric, so (remembering the Section 2 callout) convolution and correlation are identical here — no flip to worry about.

Here is a concrete 3×3 Gaussian kernel (a common approximation for small σ). Up to the normaliser it is the integer grid [1, 2, 1 / 2, 4, 2 / 1, 2, 1]. Those nine numbers add to 16, so dividing every entry by 16 makes them sum to 1. Notice the structure: the centre weight is 4/16, the four edge neighbours are 2/16 each, and the four diagonal corners (which are farther away) are only 1/16 each. Contrast this directly with the mean filter, where all nine were a flat 1/9: the Gaussian still smooths, but it lets the centre pixel keep the loudest voice, so edges are preserved noticeably better than under the brutal democracy of the box filter.

Smoothing as a first taste of denoising

Step back and notice what we have really built. The main everyday job of smoothing is denoising — cleaning up the random speckle that creeps into images. You have seen it: photos taken in dim light look grainy, peppered with little flecks of brightness and darkness that were not in the real scene. That grain is image noise, the random error a sensor adds when there is too little light for it to be confident about each pixel. Because noise is random and uncorrelated from pixel to pixel, while real scene structure is smooth and consistent across neighbours, a blur that averages neighbours together naturally suppresses the noise more than it touches the signal.

But we have also met the catch, and it is worth stating plainly because it drives the entire next stretch of this track. Both the mean and the gaussian blur are blind smoothers: they average a window regardless of what is in it. So they buy you cleaner flat regions at the cost of softer edges — every blur trades noise for sharpness. Turn σ up to kill more grain and watch the edges melt; keep σ low to protect the edges and watch the grain survive. There is no setting of a plain blur that removes noise while leaving genuine edges crisp, because the filter literally cannot tell the two apart.

Everything in this guide rests on one idea you now own: the sliding window of convolution. A small grid of weights marches across the image; overlap, multiply, add; pad the borders; choose the weights and you choose the effect — flat weights for a mean blur, a bell of weights for a Gaussian. That same sliding-window machine is what edge detection (the next-but-one guide) and even the convolutional neural networks of the later tracks are built on. Smoothing was simply the gentlest, most visual place to learn it. Next, we make these filters smart enough to respect the edges they have so far been melting away.