Matching by Looking: Templates, Colour, and the Recognition Pipeline

What does it mean to 'recognize' an image?

You do it thousands of times a day without noticing. You glance at a photo and know it is a beach. You scan a crowded train platform and instantly find your friend's face. You hold up your phone and it unlocks because it is sure the face in front of it is yours. All three are acts of recognition — turning raw light into a confident statement about the world — and getting a computer to do them is the central problem of this whole track. Before we can build a machine that recognizes anything, we have to be precise about what 'recognize' even means, because the word secretly hides three different jobs.

The first job is classification: what is in this image? You answer with a single label for the whole picture — 'cat', 'beach', 'this is a stop sign' — like dropping each photo into exactly one album. The second is detection: what is where? Now you must say both the label and the location, drawing a box around each thing — 'a person here, a dog there' — like spotting your friend's face among hundreds and pointing right at it. The third is identification and verification: which specific instance? Classification only cares that it is a face; identification asks whose face, and verification asks the yes/no question 'is this the same person as the stored one?' — exactly what your phone does when it unlocks. The same image can be fed to all three jobs and give three different kinds of answer.

So how do you build a recognizer by hand? The rest of this guide answers that by constructing the very simplest one from scratch — so simple you could do it with a sheet of tracing paper — and by laying out the shared assembly line, the classical recognition pipeline, that every method in this track plugs into. We will see exactly where it works, where it breaks, and why those cracks are the reason every later guide exists.

The classical recognition pipeline

Almost every classical recognizer is built like a factory assembly line: the raw image enters at one end, and a string of stations each do one small, well-defined job, passing the result to the next. This shared blueprint is the classical recognition pipeline, and once you can see it, every method in this track stops looking like a random trick and starts looking like a different choice for one particular station. Let us walk the line end to end.

1. Raw image — the grid of pixel numbers straight from the camera or file.
2. Preprocessing — clean it up: resize to a standard size, denoise, and normalize lighting so two photos of the same thing start out comparable.
3. Feature extraction — turn the mass of pixels into a compact descriptor that captures what matters (edges, colours, textures) and throws away what does not.
4. Representation — pack that descriptor into a fixed-length vector: an ordered list of numbers of the same size for every image, so they can be compared.
5. Matcher or classifier — compare that vector to stored examples or to a learned rule, and produce a score for each candidate answer.
6. Decision — pick the winning answer (and decide whether you are confident enough to commit to it at all).
7. Post-processing (optional) — tidy the output: merge overlapping detection boxes, smooth results over a video, reject low-confidence calls.

Why does each station earn its place? Take a concrete job: recognizing a company logo printed on products coming down a conveyor belt. Skip preprocessing and a product photographed under a yellow warehouse lamp will look nothing like your reference shot, sinking everything downstream. Skip feature extraction and you are forced to compare millions of raw pixels directly — slow, and fooled by the tiniest shift. The point of features is to summarize: the logo's distinctive corners and red colour survive, while the irrelevant background noise is dropped. The representation must be a fixed-length vector because a comparison rule needs every input to have the same shape — you cannot subtract a list of 900 numbers from a list of 12. The matcher is where the actual recognizing happens, and the decision is what protects you from confidently shouting 'logo!' at a blank box. Each station exists to make the next one's job possible.

With this scaffold in mind, the whole track becomes a tour of brilliant choices for the feature-and-matcher stations. The simplest possible choice — the one we build next — barely has a feature stage at all: it compares the raw pixels directly. That is template matching, and it is the perfect place to start because its strengths and its spectacular failures will motivate every refinement that follows.

Template matching: slide a picture and score the fit

Imagine you have a small transparent stencil printed with a picture of the thing you are looking for — say, the company logo. You lay it on top of a big photo and slide it around. At each spot you ask: how well does the stencil line up with what is underneath it? Where the overlap is best, that is where the logo is. That is the entire idea of template matching. The stored stencil is called the template; the big photo is the search image; and 'how well it lines up' is a number we will compute, called the similarity score.

1. Store the template: a small patch of pixels — the picture of what you want to find.
2. Place the template at a candidate location in the search image, lining up its top-left corner with that spot.
3. Compute a similarity score there by comparing the template's pixels with the image pixels underneath it.
4. Move one pixel over and repeat — sweep across every possible location, building a whole map of scores.
5. Return the location of the best score: that is where the template fits best.

Notice what step 2-and-4 really are: placing the template at every location and scoring each one is exactly a sliding window. We slide a fixed-size window across the whole image and evaluate it everywhere. Hold on to this phrase — in Guide 3 the sliding window becomes the backbone of real-time face detection, and the only thing that will change is what we compute inside the window. Here, what we compute inside is the simplest thing imaginable: a direct pixel comparison. Let us make 'how well it lines up' precise. This is your first vision formula, so we will go slowly.

Sum of squared differences — the most direct possible match score.

Read this as: 'to score the fit at position (u,v), look at every pixel of the template, find the gap between it and the image pixel sitting underneath it, square that gap, and add all the squares up.' Symbol by symbol: I is the search image, so I(u+x, v+y) is the brightness of one image pixel; T is the template, so T(x,y) is the matching template pixel. The pair (u,v) is the candidate location — where we have placed the template's top-left corner. The pair (x,y) is an offset inside the template that ranges over all its pixels, and the big Σ means 'sum over all those offsets.' Because differences are squared, every term is positive (mismatches never cancel out), and a small SSD means a good match — zero would be a pixel-perfect overlap. Squaring also punishes a few large mismatches far more than many tiny ones, so one badly wrong region wrecks the score. Tiny example: a 2×2 template of all 10s laid over an image patch [[12, 9],[11, 10]] gives differences 2, -1, 1, 0, whose squares 4, 1, 1, 0 sum to SSD = 6 — a near-perfect fit. Slide it onto a bright patch of all 50s and the differences are 40 each: SSD = 4×1600 = 6400. The huge number screams 'this is not it.'

Normalized cross-correlation — a brightness- and contrast-proof match score.

SSD has a fatal flaw: turn the lights up and every image pixel rises by, say, 30, so every difference balloons and SSD declares 'no match' even though it is the very same scene. Normalized cross-correlation (NCC) fixes this in two moves. First, mean-subtraction: Ī(u,v) is the average brightness of the image patch under the current window, and T̄ is the average brightness of the template. Subtracting each mean throws away the overall brightness and keeps only the pattern of light and dark — the relative ups and downs. Second, magnitude-normalization: the denominator is the square root of (the patch's total squared variation times the template's), which rescales both to unit size so contrast no longer matters. The numerator is then pure alignment: it is large and positive when bright spots in the template line up with bright spots in the image and dark with dark. Because of the normalization, NCC always lands between −1 and 1: a value of 1 is a perfect match (identical pattern, any brightness or contrast), 0 means unrelated, and −1 means a perfect photographic negative. Concretely, brighten our earlier patch uniformly by 30 — every pixel and the mean rise together, the subtraction cancels the +30 exactly, and NCC stays at 1, declaring the match SSD would have missed.

# Template matching by sum of squared differences (SSD).
# I: the search image (H x W); T: the template (h x w). Grayscale for clarity.
def template_match_ssd(I, T):
    H, W = I.shape
    h, w = T.shape
    best_score = float("inf")        # SSD: smaller is better, so start as high as possible
    best_loc = (0, 0)
    # Sweep the template's top-left corner over every valid location (u, v).
    # This double loop IS the sliding window.
    for v in range(H - h + 1):
        for u in range(W - w + 1):
            patch = I[v:v + h, u:u + w]     # the window under the template
            diff = patch - T               # per-pixel difference
            score = (diff * diff).sum()    # sum of squared differences
            if score < best_score:
                best_score = score
                best_loc = (u, v)          # remember the best fit so far
    return best_loc, best_score            # where (and how well) the template fits best

Why raw template matching falls apart

That little recognizer is real and genuinely useful — for finding a fixed icon on a screen, lining up two scans of the same document, or spotting a known part on an assembly line, it is fast and reliable. But the moment the world is allowed to vary even slightly, it crumbles, and each way it crumbles is a signpost pointing to a later guide in this track. Let us face the failure modes honestly.

Scale is the first killer. Your template is, say, 60 pixels wide. If the object appears 90 pixels wide because it is closer to the camera, the pixels no longer line up at all and the score collapses — even though it is plainly the same object. Rotation and viewpoint are just as brutal: tilt the object 30 degrees, or photograph the same chair from the side instead of the front, and pixel-for-pixel comparison falls to pieces. A flat stencil has no idea that a rotated or resized object is 'the same'. The usual patch is to keep many templates — several sizes, several angles — and try them all, but the cost explodes and you can never store every possibility.

Lighting is partly tamed — NCC already shrugged off a uniform brightness change and an overall contrast change. But it does not save you from a shadow falling across half the object, from a coloured light casting an orange tint, or from a glossy highlight. These change the pattern of brightness, not just its overall level, and pattern is exactly what NCC trusts. Occlusion is worse still: hide part of the object behind a coffee mug and many template pixels now sit over the mug, injecting big errors that drag any pixel-wise score down even though most of the object is in plain sight.

But the deepest failure — the one that ends the whole template-matching dream — is intra-class variation. A template is a photo of one specific thing. A template of one particular chair can, with luck, find that exact chair again. It can never recognize chairs as a category, because office chairs, stools, armchairs and beanbags share almost no pixels in common. The same is true of faces, cats, or handwritten 7s: a category is a vast cloud of appearances, and no single stored example sits at its centre. Template matching compares to one example and only one example.

Recognizing by colour: histograms

Template matching obsessed over where every pixel sits, which is exactly why a small shift or rotation destroyed it. So let us try the opposite extreme and throw away position entirely. Instead of asking where the colours are, ask only how much of each colour there is. A ripe-tomato photo is mostly red with a little green, no matter where the tomato sits in the frame or which way it is turned. That count of 'how much of each colour' is a colour histogram — think of it as a colour fingerprint of the image — and comparing fingerprints is the heart of colour histogram matching.

Building one is a simple tally. Recall that each colour pixel is three numbers — red, green, blue, each 0 to 255. We do not want a separate slot for all 16.7 million possible colours, so we bin: chop each channel into, say, 8 coarse ranges, giving 8×8×8 = 512 colour bins. Then we walk every pixel, decide which bin its colour falls in, and add one to that bin's count. Finally we divide every count by the total number of pixels so the bars sum to 1 — now images of different sizes are comparable. The payoff is automatic invariance: because we only counted colours and never recorded positions, sliding the object across the frame, rotating it, or slightly bending it does not change the histogram at all. The very thing that broke template matching simply does not register here.

The weakness is the mirror image of that strength: by discarding position, a histogram cannot tell a red square on blue from blue on red, and a tablecloth and a tomato that share the same red could be confused. It is also sensitive to illumination — change the lighting and the colours themselves shift bins, scrambling the fingerprint. The standard remedy is to compute histograms in a colour space that separates colour from brightness (for example using hue, or red/green ratios), so the fingerprint survives a dimmer room. Despite the limits, colour histograms drove an entire generation of practical systems: content-based image retrieval ('find me more pictures like this one') and colour-based tracking, where the CamShift algorithm follows a coloured object through a video by chasing its histogram frame to frame. To use any of this we need one more thing: a way to score how alike two fingerprints are.

Histogram intersection — the amount of colour two images share.

Read this as: 'go bin by bin, take whichever histogram has less in that bin, and add up all those minimums.' Symbol by symbol: H₁ and H₂ are the two normalized histograms we are comparing (each summing to 1); i is the bin index, running over all colour bins; H₁,ᵢ and H₂,ᵢ are the heights of bin i in each. Why the minimum? It measures overlap. In each bin, the shared amount can be no more than the smaller of the two — like pouring two stacks of coins into the same column and keeping only the part where they both reach. Summing over all bins gives the total shared colour: the result runs from 1 (identical fingerprints — every bin matches exactly) down to 0 (completely disjoint — wherever one has colour, the other has none). Worked example: H₁ = [0.5, 0.3, 0.2] and H₂ = [0.4, 0.4, 0.2]. The per-bin minimums are 0.4, 0.3, 0.2, which sum to 0.9 — the two images share 90% of their colour content, a strong match.

Chi-square distance — a difference measure that weights rare colours fairly.

Intersection scores similarity (bigger is better); chi-square scores distance (smaller is better, 0 means identical). Read it as: 'for each bin, square the gap between the two histograms, then divide that by how much colour the bin holds in total, and sum up.' The symbols H₁, H₂, and i mean the same as before; the new twist is the denominator H₁,ᵢ + H₂,ᵢ, the combined height of the bin. Why divide by it? Without dividing, a gap of 0.1 in a giant bin (where both images are, say, 50% red) would count the same as a gap of 0.1 in a tiny bin (where one image has a rare splash of purple and the other has none). But the rare colour is far more telling — it is exactly the distinctive detail that separates two otherwise-similar pictures. Dividing by the bin total inflates the contribution of these small, rare bins, so chi-square pays attention to the discriminating colours instead of being dominated by whichever colour happens to flood the image. That is why retrieval systems often preferred it to a plain squared difference.

import numpy as np

def color_histogram(img, bins=8):
    # img: H x W x 3 array of uint8 RGB values (0..255).
    # Quantize each channel into `bins` coarse levels, then combine the three
    # channel indices into one bin number per pixel.
    q = (img.astype(int) * bins) // 256                 # each channel -> 0..bins-1
    idx = (q[..., 0] * bins + q[..., 1]) * bins + q[..., 2]
    hist = np.bincount(idx.ravel(), minlength=bins ** 3).astype(float)
    return hist / hist.sum()                            # normalize so the bins sum to 1

def intersection(h1, h2):
    return np.minimum(h1, h2).sum()                     # 1.0 = identical, 0.0 = disjoint

def chi_square(h1, h2):
    denom = h1 + h2
    safe = denom > 0                                    # skip empty bins (avoid 0/0)
    return ((h1[safe] - h2[safe]) ** 2 / denom[safe]).sum()   # 0.0 = identical

From matching to learning

Step back and notice what templates and colour histograms have in common. Both are forms of matching: to find something, you must already have the exact thing you are looking for — a stored stencil, a stored fingerprint — and you compare new images against that single stored example. They sit in the same classical recognition pipeline, differing only in the feature they extract and the score they compute. And both share the same ceiling we hit twice already: one stored example gives you no principled way to generalize across the natural variation of a whole category. Show the system a chair it has never seen and matching has nothing to compare against.

Which raises the question that powers the next guide: what if, instead of comparing each new image to one stored example, we collected many labelled examples — hundreds of chairs, hundreds of not-chairs — and let the computer learn a decision rule from the whole collection? Then 'chair-ness' is no longer one picture; it is a region of the feature space that many examples carve out together. The variation we struggled to forgive becomes the very evidence the rule is built from.