From Pixels to Features: Edges and Gradients

What Is a Feature, and Why Not Just Use Pixels?

Imagine you are meeting a friend at a busy train station. You do not find them by remembering 'they will be standing exactly 4.2 metres to the left of the clock.' People move. Instead, you scan for stable, recognizable landmarks: their red hat, their round glasses, the way their backpack sits. Those landmarks are distinctive, you can spot them from different angles, and they stay recognizable even if your friend walks across the hall. Computer vision works the same way. A feature is a distinctive, locally identifiable spot in an image — a place whose appearance is special enough that you could re-find the very same spot in another photo of the same scene.

A feature location worth remembering is called a image keypoint — informally, 'a point the algorithm decides to write down and come back to.' Think of keypoints as the pins you would drop on a map: not every patch of road, just the memorable intersections and landmarks. The whole job of the first half of a vision pipeline is choosing good pins and writing a short, robust note describing what each one looks like, so that later we can recognize the same pins in a different image.

So why can't we just use the raw pixel values themselves as identity? Because a pixel's number is shockingly unstable. The very same physical point — say, the corner of a window — produces totally different brightness numbers depending on the lighting (sunny vs. cloudy), the viewpoint (you moved left), the scale (you stepped closer), and plain sensor noise. Two photos of the same building can have pixel value 47 at that corner in one and 180 in the other. If we tried to match images by comparing raw pixel numbers, almost nothing would line up. We need descriptions that survive these changes — and that begins with looking not at brightness itself, but at how brightness changes across the image.

Here is the mental map for this whole track. Almost every classical vision pipeline runs in three stages: detect → describe → match. Detect finds the keypoints worth remembering. Describe turns the neighbourhood around each keypoint into a compact vector of numbers that is stable under lighting and viewpoint changes. Match compares those descriptions across two images to say 'this pin here is the same as that pin there.' This very first guide lays the foundation under all three by introducing gradients and edge detection — the raw signal of 'where is brightness changing?' that every later detector and descriptor is built on.

The Image as a Landscape of Brightness

Let's pin down what an image is, mathematically, because the rest of the track leans on this picture. A grayscale image is a function. Give it a location — a column x and a row y — and it returns one number: the brightness at that pixel, usually an integer from 0 (black) to 255 (white). We write it as I(x, y). The whole image is just this function evaluated at every integer (x, y) on a grid. There is nothing mysterious here: 'the image' and 'a table of brightness numbers indexed by position' are the same object.

Now picture that table as a physical landscape. Walk over the image grid and treat brightness as height: bright pixels are high ground, dark pixels are low ground. A photo of a white wall is a flat plateau. A photo with a dark doorway in a bright wall has a tall plateau with a deep rectangular pit punched into it. The edges of that doorway are cliffs — steep drops where height plunges over a tiny horizontal distance. This 'image = terrain' analogy is worth holding onto, because it makes the central insight obvious.

The insight: all the information that makes a location identifiable lives where the landscape changes — on the slopes, ridges, and peaks — not where it is flat. A flat plateau is featureless; one spot on it looks exactly like its neighbour, so you could never tell them apart or re-find one of them. But a cliff edge, a sharp ridge, or an isolated peak is distinctive. This is precisely why features begin with measuring change in brightness. Flat = no information. Steep = potential feature.

Make it concrete with two tiny 3×3 patches of pixels. On the left, a flat region where every value is 100 — perfectly uniform, no change in any direction. On the right, a vertical brightness step: the left two columns are dark (30) and the right column jumps to bright (220). Reading across any row of the right patch, brightness leaps by 190 in one step. That leap is the cliff. The flat patch hides a feature nowhere; the step patch screams 'edge here.'

# Two 3x3 brightness patches (rows top-to-bottom)

flat = [[100, 100, 100],     # every value identical ->
        [100, 100, 100],     # no change in any direction ->
        [100, 100, 100]]     # featureless plateau

step = [[ 30,  30, 220],     # dark, dark, BRIGHT  -> a jump of 190
        [ 30,  30, 220],     # same jump on every row
        [ 30,  30, 220]]     # this vertical cliff IS an edge

# Reading left-to-right across a row:
#   flat: 100 -> 100 -> 100   (change = 0, nothing here)
#   step:  30 ->  30 -> 220   (change = +190 between cols 2 and 3)

Gradients: Measuring Change

We have decided that 'change in brightness' is the raw material of features. Now we measure it precisely. Ask two simple questions at every pixel: *how fast does brightness change as I take one step to the right?* and *how fast does it change as I take one step down?* The answer to each is a slope — exactly like the steepness of our terrain. Bundle the two slopes together and you get the image gradient: a small arrow at every pixel that points in the direction brightness increases fastest, and whose length tells you how steep that increase is.

How do we estimate a slope from a grid of numbers? With a finite difference — the discrete cousin of a derivative. The cleanest version is the central difference: to find the rate of change in the x-direction at a pixel, subtract the brightness one step left from the brightness one step right. Big difference means a steep slope; zero means flat. Do the same vertically for the y-direction. Stack those two into a vector and you have the gradient.

The image gradient: a pair of slopes, one along x, one along y.

Let's unpack every symbol. I is the brightness function from the last section — I(x, y) is the brightness at column x, row y. ∇I (read 'grad I') is the gradient: a two-component vector at each pixel. I_x is the horizontal rate of change — how much brightness rises per step to the right — and we estimate it as the brightness one column to the right, I(x+1, y), minus the brightness one column to the left, I(x−1, y). I_y is the vertical rate of change, estimated the same way using the rows above and below. A worked number: take our step patch and look at the centre pixel. To its right sits 220, to its left sits 30, so I_x ≈ 220 − 30 = 190 — a strong horizontal change. Above and below it are both 30, so I_y ≈ 30 − 30 = 0 — no vertical change. The gradient there is (190, 0): a strong arrow pointing right, from dark toward bright.

From the two slopes we read off edge strength (magnitude) and edge orientation (angle).

The gradient vector carries two pieces of information we will reuse constantly. Its magnitude |∇I| = √(I_x² + I_y²) is just the length of the arrow (the Pythagorean combination of the two slopes); it measures edge strength — how steep the brightness cliff is here. Its orientation θ = atan2(I_y, I_x) is the angle the arrow points; it measures edge direction, the way brightness increases most steeply. (atan2 is the safe two-argument arctangent that returns the correct angle in all four quadrants instead of getting confused by signs.) Plugging in our centre pixel: magnitude = √(190² + 0²) = 190 (a strong edge), and θ = atan2(0, 190) = 0 radians (the arrow points straight right). Crucially, the edge itself runs perpendicular to the gradient — the doorway's vertical edge runs up-down while its gradient points sideways. Hold onto magnitude and orientation: descriptors like SIFT and HOG, two guides from now, are built almost entirely from histograms of exactly these two quantities.

The plain central difference works, but it has a flaw: it looks at just two pixels and so it amplifies noise — one stray speck makes a fake slope. The fix used everywhere in practice is the Sobel operator, which estimates the same derivatives but averages over a 3×3 window so a single noisy pixel can't dominate. And the way it scans that window over the image is exactly convolution, the sliding-weighted-window operation you met in earlier tracks: place a small grid of weights (the kernel) on top of each pixel, multiply each overlapping pair, sum them up, and write the result at the centre. Slide and repeat across the whole image. Sobel is just convolution with two cleverly chosen kernels. This is also why the next guides, and edge detection in general, are framed in terms of convolutions.

The two Sobel kernels: G_x estimates the horizontal derivative I_x, G_y the vertical derivative I_y.

Read G_x column by column and you can see it both smooth and differentiate at once. Left-to-right it is (−1, 0, +1): a central difference (right minus left) — that is the differentiating part. Top-to-bottom the weights are (1, 2, 1): a little weighted average that blurs vertically, with the centre row counted double — that is the smoothing part. So G_x answers 'is there a vertical edge here?' while quietly averaging over three rows to shrug off noise. G_y is the same kernel rotated 90°: it differentiates top-to-bottom and smooths left-to-right, answering 'is there a horizontal edge here?' Convolve the image with G_x to get I_x and with G_y to get I_y, then feed them into the magnitude and orientation formulas above. Let's verify on the step patch: convolving G_x with that 3×3 gives (−1·30 +1·220) + (−2·30 +2·220) + (−1·30 +1·220) = 190 + 380 + 190 = 760 — large, a clear vertical edge. Convolving G_y gives (−1·30 −2·30 −1·220) + (+1·30 +2·30 +1·220) = 0 — no horizontal edge. Exactly the answer our intuition demanded.

import numpy as np

def gradients(I):
    """Estimate gradient magnitude and orientation from a grayscale image I."""
    I = I.astype(float)
    Ix = np.zeros_like(I)
    Iy = np.zeros_like(I)

    # Central differences (ignore the 1-pixel border for simplicity).
    Ix[1:-1, 1:-1] = I[1:-1, 2:] - I[1:-1, :-2]   # right minus left  -> I_x
    Iy[1:-1, 1:-1] = I[2:, 1:-1] - I[:-2, 1:-1]   # below minus above -> I_y

    magnitude   = np.sqrt(Ix**2 + Iy**2)          # edge strength |grad I|
    orientation = np.arctan2(Iy, Ix)              # edge direction theta (radians)
    return magnitude, orientation

# In practice you would convolve with the Sobel kernels instead of bare
# differences, because Sobel also smooths and is far less noise-sensitive:
#   Gx = [[-1,0,1],[-2,0,2],[-1,0,1]]
#   Gy = [[-1,-2,-1],[0,0,0],[1,2,1]]
#   Ix = convolve2d(I, Gx);  Iy = convolve2d(I, Gy)

Edges: The First Feature

Now we can define our first real feature. An edge is a location where brightness changes rapidly — in our terms, a pixel where the gradient magnitude |∇I| is large. Edges trace the outlines of objects, the boundaries between light and shadow, the contours of letters on a page. They are the skeleton of an image: even a pure edge map, with all the smooth interiors thrown away, is usually enough for a human to recognize the scene. Detecting them is the classic task of edge detection.

The simplest possible edge detector follows directly from the last section: compute the gradient magnitude everywhere, then keep only the pixels where it exceeds some threshold. Below the threshold, 'flat enough, not an edge'; above it, 'steep enough, call it an edge.' That is the entire algorithm — three lines of code — and it genuinely does light up the boundaries in an image.

def simple_edges(I, threshold):
    """Naive edge detector: threshold the gradient magnitude."""
    magnitude, _ = gradients(I)
    edge_map = magnitude > threshold   # True where brightness changes steeply
    return edge_map

# Pick threshold too LOW  -> noise and texture flagged as edges (speckle).
# Pick threshold too HIGH -> faint but real edges vanish (broken outlines).
# The 'right' value depends on the image, the lighting, even the region.

Try it on a real photo, though, and the cracks show immediately. First, the edges come out thick: a real cliff in brightness is several pixels wide, so a whole band of pixels clears the threshold and you get a fat smear instead of a clean one-pixel line. Second, they come out broken: along a single contour the gradient strength fluctuates, dipping below the threshold here and there, so a continuous edge shatters into dashes. Third, they come out speckled with noise: random sensor fluctuations produce small gradients everywhere, and if your threshold is low enough to catch faint real edges, it also catches a blizzard of fake ones. And worst of all, the right threshold is not a universal constant — it depends on the image, the lighting, and even which region you are looking at. A value that cleans up the bright sky over-suppresses the dim foreground.

The Canny Edge Detector: A Careful Pipeline

John Canny's 1986 edge detector is the classic answer to every weakness we just listed, and it is still a workhorse today. Its genius is not one clever trick but a careful ordering of five stages, each one fixing a specific failure of naive thresholding. Read it as a pipeline: the image flows in one end, and a clean map of thin, connected, single-pixel-wide edges comes out the other.

1. Gaussian smoothing — blur the image gently first, so that random noise is averaged away before we ever take a derivative. This is the single most important fix for the 'speckle' problem.
2. Gradient computation — run Sobel on the smoothed image to get the magnitude |∇I| and orientation θ at every pixel, exactly as in Section 3.
3. Non-maximum suppression — thin the thick edge bands down to one-pixel ridges by keeping a pixel only if it is the local maximum of magnitude along the gradient direction.
4. Double thresholding — classify each surviving pixel as strong (clearly an edge), weak (maybe an edge), or none, using two thresholds instead of one.
5. Hysteresis — keep every strong pixel, and keep a weak pixel only if it is connected to a strong one. This stitches broken edges back together while rejecting isolated noise.

Stage 1 deserves the most attention because the ordering is the whole point: we smooth before we differentiate. The smoothing is done by convolving with a 2-D Gaussian, the bell-shaped weight kernel below. It is the principled way to average each pixel with its neighbours, weighting nearby pixels more than distant ones.

The 2-D Gaussian smoothing kernel used in stage 1 of Canny.

Symbol by symbol: x and y are offsets from the centre of the kernel — (0, 0) is the pixel we are computing, (1, 0) is its right neighbour, and so on. G(x, y) is the weight given to the neighbour at that offset. σ (sigma) is the standard deviation, the single knob that controls the blur radius: a small σ blurs only the nearest pixels (preserves fine detail but suppresses less noise), while a large σ spreads the averaging over a wider neighbourhood (very smooth, fewer tiny edges survive). The exp(−(x² + y²)/(2σ²)) term is the bell curve: x² + y² is the squared distance from the centre, so the weight falls off smoothly the farther a neighbour sits — a pixel right next door counts far more than one three steps away. The leading 1/(2πσ²) is a normalizing constant chosen so that all the weights sum to 1; this matters because it means smoothing does not brighten or darken the image overall — it only redistributes brightness locally. For example, with a tiny 3×3 kernel the centre weight might be about 0.25 and each of the four edge-neighbours about 0.12, the four corners less still — and those nine numbers add up to 1.

Why blur before differentiating? Because differentiation amplifies noise, and the order matters enormously. A derivative responds to change, and noise is change — fast, random, pixel-to-pixel jitter. Differentiate a noisy image directly and every speck becomes a sharp spike, drowning the real edges in false ones. Smoothing first gently averages out that random jitter (which has no consistent direction and so cancels), while a true edge — a coherent cliff many pixels long — survives the blur almost intact. Then differentiating the smoothed image responds to the real cliffs, not the noise. Smooth-then-differentiate keeps the signal and kills the noise; differentiate-then-smooth would have already let the noise explode.

Stage 3, non-maximum suppression, is the cure for thick edges and is worth picturing concretely. Recall that a real edge produces a band of high-magnitude pixels, several wide, all clustered around the true cliff. We want just the crest of that band. So at each pixel we look along its gradient direction θ — straight across the edge, the direction of steepest change — and compare its magnitude to the two neighbours immediately ahead and behind in that direction. If the centre pixel is the local maximum (strictly greater than both), it sits on the crest, so we keep it. If either neighbour is larger, the centre is on the band's slope, not its peak, so we suppress it to zero. Picture magnitudes 30, 90, 50 across an edge: only the 90 is a local max, so 30 and 50 are zeroed and a 3-pixel-wide band collapses to a clean 1-pixel ridge. Do this everywhere and the fat smears become crisp lines. Stages 4 and 5 then handle the broken-and-speckled problem: instead of one threshold, two define strong and weak edges, and edge detection is finalized by hysteresis, which trusts a weak pixel only when it chains onto a strong one — reconnecting dashed contours while discarding lonely noise specks. This careful pipeline is exactly what the canny edge detector does, and why it beats naive thresholding so decisively.

From Edges to Keypoints: Why Edges Aren't Enough

We now have beautiful, clean edges. So are we done — can we use edge points as the keypoints we match between images? Not quite, and the reason is subtle and important enough that it drives the entire next guide. The problem is that an edge point cannot be located precisely. It tells you where a boundary is, but not exactly which point along that boundary you are looking at.

Run a thought experiment — the famous aperture problem. Cut a small hole (an 'aperture') in a card and look at an image through it, so you see only a little circular window. Place that window squarely on a long, straight edge — say the vertical boundary of a dark door against a bright wall. Now slide the window a little along the edge, parallel to it. What do you see? Exactly the same thing: a vertical boundary cutting through your window, identical before and after. The view is unchanged, so you have no way to tell how far you slid. The edge has 'erased' all information about position along its own length.

Connect this back to gradients and it makes complete sense. Along a straight edge, the gradient is strong in one direction (perpendicular to the edge) and essentially zero in the other (parallel to it). Because there is no change parallel to the edge, motion in that direction is invisible. So an edge pins down only the perpendicular offset — how far across the boundary you are — and leaves the parallel offset completely unknown. An edge point is localizable in one direction but ambiguous in the other. For matching two images, that one-dimensional ambiguity is fatal: you could never confidently say 'this exact spot here is that exact spot there.'

The escape is to demand more. We want locations that are localizable in both directions at once — points where, no matter which way you nudge that little aperture window, the view changes. Those are corners: places where two edges meet, so there is strong gradient in two different directions and any slide is detectable. A corner can be pinned down to a single pixel in 2-D, which is exactly what a stable, matchable image keypoint needs. (Later we will also want features that carry a notion of size or scale — round 'blob' regions — but corners come first.)

That is the bridge into the rest of the track. We began with raw pixels, learned they are unstable, and replaced them with gradients — the language of change. From gradients we built edges, our first feature, and refined them into clean contours with Canny. But edges suffer the aperture problem, so the next guide turns to corners and the harris corner detector, which measures gradient change in two directions at once to find points you can truly pin down. After that comes scale and blobs, then descriptors (SIFT, HOG) that summarize a keypoint's neighbourhood using exactly the gradient magnitude and orientation you learned here — and finally matching, where all the pieces detect → describe → match snap together.