Optical Flow: Measuring Motion Pixel by Pixel

What optical flow really is

In the previous guide we watched pixels shift between frames and got the vague feeling that 'something moved.' Optical flow is what turns that feeling into a number. Formally, optical flow assigns to every pixel a small 2D vector (u, v): u is how many pixels the brightness pattern at that location appears to slide horizontally between two consecutive frames, and v is how far it appears to slide vertically. Stack one such vector at each pixel and you get a dense flow field — a map of motion across the whole image. This is the most basic and most reusable form of motion estimation: a vector everywhere, not just a label saying 'moving' or 'still'.

There are two standard ways to visualize this field. The first is a grid of little arrows: at sampled pixels we draw an arrow whose direction is (u, v) and whose length is the speed. The second is a colour-coded flow map: we map the direction of (u, v) to a hue (say, rightward = blue, upward = green) and the speed to saturation (faint where still, vivid where fast). The colour map is how almost every modern flow result is shown, because a million tiny arrows would be unreadable.

Now the crucial subtlety. Optical flow measures apparent motion — what the brightness pattern does on the image plane — which is not always the same as the true 3D motion of the world. Think of markings drifting on the surface of a river: the patterns you see slide downstream are real, but they tell you about the water's surface, not necessarily about a stick frozen on the riverbed beneath. Two classic counterexamples make this unforgettable. A spinning barber pole turns horizontally, yet the diagonal stripes appear to crawl straight up — the apparent motion points the wrong way. And a perfectly featureless white wall can rush past the camera while producing zero apparent motion, because nothing in the brightness changes from frame to frame. Flow sees brightness, not matter.

The brightness constancy assumption

To compute flow we need one physical assumption to anchor everything, and it is beautifully simple: when a small patch of the scene moves, its brightness stays the same — it just lands somewhere new. A red dot that was bright at one spot is still equally bright a moment later, only shifted by (u, v). This is the brightness constancy assumption, and although the world breaks it sometimes (lighting changes, shadows, shiny surfaces), it holds well enough over the tiny time gap between two adjacent video frames to be the foundation of classical optical flow.

Brightness constancy: the intensity at pixel (x, y) at time t equals the intensity at the shifted location one frame later.

Let us read that equation slowly. I is the image intensity (brightness) — a function that takes a position and a time and returns a number, say 0 for black up to 255 for white. (x, y) is the pixel we are looking at, t is the current time, and dt is the tiny time step to the next frame. The unknowns u and v are the horizontal and vertical displacement of that patch during dt — exactly the flow we want. The equation simply states the assumption in symbols: the brightness I saw here-and-now equals the brightness I will find at the moved-to location (x+u, y+v) a frame later. Concretely, if a bright patch with value 200 sits at (10, 10) and the equation holds with (u, v) = (3, -1), then one frame later the pixel at (13, 9) also reads 200.

The equation above is true but useless as written — the unknowns are buried inside I. The trick is that between adjacent frames the motion is small, so we can replace the right-hand side with a first-order Taylor expansion around (x, y, t). 'First-order' just means we keep the linear terms — the value plus a rate-of-change times a small step — and throw away the higher-order curvature terms, because when u, v, and dt are tiny their squares are negligibly tinier. Expanding gives I(x+u, y+v, t+dt) ≈ I(x, y, t) + I_x·u + I_y·v + I_t·dt. Substitute that back into brightness constancy, the I(x,y,t) cancels on both sides, divide through by dt, and you arrive at the centerpiece of this whole guide:

The Optical Flow Constraint Equation (the 'brightness constancy' in differential form).

This compact line is the Optical Flow Constraint Equation, and every symbol earns its place. I_x = ∂I/∂x and I_y = ∂I/∂y are the spatial gradients: how fast brightness changes as you step one pixel sideways (I_x) or up/down (I_y). They are large at edges and near zero on smooth, flat patches — they answer 'how textured is it here?' I_t = ∂I/∂t is the temporal gradient: how much this exact pixel's brightness changed between the two frames — it answers 'how much did things move past me?' And (u, v) is still the unknown flow. Read in plain words: the equation says the brightness change you measure over time (I_t) must be exactly accounted for by the motion (u, v) sliding the spatial pattern (I_x, I_y) underneath the pixel. As a quick sanity check, if a region is perfectly flat then I_x = I_y = 0, the equation collapses to I_t = 0, and it tells you nothing about (u, v) — which is precisely the white-wall problem from Section 1.

The aperture problem

Here is the under-determination made vivid. Imagine watching a long straight edge — the border between a black bar and a white background — through a tiny hole, an aperture, so small you cannot see the ends of the edge. Now the bar slides. You can clearly tell it is moving perpendicular to the edge (the boundary creeps across your peephole), but you have absolutely no way to detect any motion along the edge, because sliding a featureless straight line along its own length looks identical to it standing still. The component of motion parallel to the edge is genuinely invisible through the aperture. This is the aperture problem.

This is not a fuzzy intuition — it is exactly what the constraint equation told us. Rewrite I_x·u + I_y·v + I_t = 0 as a dot product: ∇I · (u, v) = −I_t, where ∇I = (I_x, I_y) is the spatial gradient, which always points across the edge (in the direction of steepest brightness change). The equation pins down only the component of (u, v) along ∇I — that is, the motion perpendicular to the edge, called the normal flow. Any amount of motion perpendicular to ∇I (i.e. along the edge) adds nothing to the dot product and so satisfies the equation just as well. One equation fixes one component and leaves the other completely free. The aperture problem and the 'one equation, two unknowns' counting are the same fact wearing two costumes.

The barber pole is the showpiece of this illusion. The pole rotates about its vertical axis, so the physical motion of its surface is purely horizontal. But the painted stripes run diagonally, and through your eyes' effective aperture you only perceive each stripe's motion perpendicular to itself — which points upward. Hence the famous, maddening illusion of endless upward-climbing stripes on a pole that is merely spinning sideways. Apparent motion (up) and true motion (sideways) part ways precisely because of the aperture problem.

Lucas-Kanade: assume neighbors move together

The Lucas-Kanade method makes the simplest possible extra assumption and gets enormous mileage from it: every pixel inside a small window moves with the same flow (u, v). Picture a 5×5 patch around the pixel of interest. One pixel gave us one constraint equation and left us stuck. But if all 25 pixels in the window share the same (u, v), then each contributes its own copy of the constraint equation — with its own I_x, I_y, I_t — yet all about the same two unknowns. Twenty-five equations for two unknowns is over-determined, and over-determined linear systems have a standard best-fit answer: least squares. The aperture problem dissolves as soon as the window contains gradients pointing in more than one direction.

Stack the window's equations into matrix form A·v = −b, where each row of A is one pixel's gradient (I_x, I_y), the vector b holds that pixel's temporal gradient I_t, and v = (u, v) is the shared flow. There is generally no exact solution (the data is noisy), so we minimize the squared residual ‖A·v + b‖². Setting the derivative to zero gives the normal equations, which expand into a tidy 2×2 system built from sums of gradient products over the window:

Lucas-Kanade least-squares solution; the 2×2 matrix on the left is the structure tensor (second-moment matrix), all sums taken over the window.

Let us decode each piece. The left matrix, A⊤A = [[ΣI_x², ΣI_xI_y], [ΣI_xI_y, ΣI_y²]], is the structure tensor (or second-moment matrix). Each entry is a sum over the window: ΣI_x² measures how much horizontal-edge energy the window contains, ΣI_y² the vertical-edge energy, and ΣI_xI_y how the two correlate. The right side, −(ΣI_xI_t, ΣI_yI_t), couples the spatial gradients to the temporal one — it is the 'how-much-and-which-way did brightness change in time' term. Solve this little 2×2 system (invert the matrix, multiply) and out pops the single flow vector (u, v) for the whole window. To make it concrete: a textured corner patch might give A⊤A = [[800, 50], [50, 600]] and right-hand side (−400, 300); inverting yields a clean, unique (u, v) because the matrix is well-conditioned (both diagonal terms large, determinant far from zero).

Here is the beautiful connection to earlier CV material: that 2×2 structure tensor is exactly the Harris corner matrix. The same quantity that decides 'is this a corner?' decides 'is the flow here trustworthy?' Its two eigenvalues tell the story. Two large eigenvalues mean gradients in two distinct directions — a corner — and the system is well-conditioned, so flow is reliable. One large and one near-zero eigenvalue means a single dominant edge direction — the aperture problem returns, the matrix is nearly singular, and only the normal-flow component is recoverable. Both eigenvalues near zero means a flat, textureless region — the matrix is essentially zero, uninvertible, and flow is hopeless (the white wall again). The aperture problem, hand-wavy in Section 3, is now a precise statement about the rank of a matrix.

import numpy as np

def lucas_kanade(I1, I2, x, y, win=5):
    """Estimate flow (u, v) for the pixel at (x, y) using a win x win window.
    I1, I2 are consecutive grayscale frames (float arrays)."""
    h = win // 2
    # Spatial gradients (from frame 1) and temporal gradient (frame diff)
    Ix = 0.5 * (I1[:, 2:] - I1[:, :-2])          # d/dx via central difference
    Iy = 0.5 * (I1[2:, :] - I1[:-2, :])          # d/dy via central difference
    It = I2 - I1                                  # d/dt: brightness change

    # Gather gradients inside the window around (x, y)
    gx = Ix[y-h:y+h+1, x-h:x+h+1].ravel()
    gy = Iy[y-h:y+h+1, x-h:x+h+1].ravel()
    gt = It[y-h:y+h+1, x-h:x+h+1].ravel()

    # Structure tensor A^T A  (== Harris corner matrix) and right-hand side
    ATA = np.array([[np.sum(gx*gx), np.sum(gx*gy)],
                    [np.sum(gx*gy), np.sum(gy*gy)]])
    ATb = np.array([np.sum(gx*gt), np.sum(gy*gt)])

    # Solve only where the window is corner-like (both eigenvalues large)
    eig_min = np.linalg.eigvalsh(ATA)[0]
    if eig_min < 1e-2:                            # flat or single-edge -> unreliable
        return None
    u, v = np.linalg.solve(ATA, -ATb)            # the normal equations
    return u, v

Sparse vs dense, and seeing big motions

Lucas-Kanade as described has a built-in limitation: it only produces trustworthy vectors at well-textured, corner-like points, so its output is sparse — a scattering of arrows on the interesting parts of the image, blanks everywhere else. That is perfect for tracking a handful of features, but many tasks want a vector at every single pixel — that is dense optical flow. To fill the flat, textureless regions where Lucas-Kanade gives up, we need a different philosophy: instead of solving each window in isolation, solve for the whole flow field at once and let confident regions propagate their answer into the uncertain ones.

The classic dense method, Horn-Schunck, encodes that philosophy as an energy (a cost) to minimize over the entire image. It has exactly two competing terms. The first is the brightness-constancy data term: the squared left-hand side of our constraint equation, which we want near zero everywhere. The second is a smoothness term that penalizes the flow for changing rapidly from one pixel to its neighbours — this is the regularizer, the 'extra assumption' the aperture problem demanded, now made explicit. Minimizing their sum lets textured pixels (where the data term is strong) dictate the flow, while the smoothness term gently carries that flow across the flat gaps.

Horn-Schunck energy: data fidelity plus a smoothness regularizer, balanced by λ.

Reading it term by term: the double integral ∬...dx dy just means 'add up the cost over every pixel in the image.' Inside, the data term (I_x u + I_y v + I_t)² is the optical-flow constraint from Section 2, squared so that any violation (positive or negative) counts as cost — minimizing it pushes the flow to obey brightness constancy. The smoothness term uses ∇u = (∂u/∂x, ∂u/∂y), the spatial gradient of the horizontal-flow field itself (and likewise ∇v); ‖∇u‖² is large wherever neighbouring flow vectors disagree, so penalizing it forces the field to vary gently. The knob λ (lambda) sets the trade-off: small λ trusts the data and gives sharp but noisy flow that crumbles in flat regions; large λ enforces smoothness and gives clean flow that unfortunately blurs across true motion boundaries (an object's edge against its background). Tuning λ is choosing how much to believe the pixels versus the assumption — for example λ ≈ 0.1 might suit clean footage, while noisier video wants a larger λ to suppress spurious vectors.

One last practical wall: every method here rests on the small-motion Taylor approximation, which breaks for fast objects. If a ball jumps 40 pixels between frames, the linearization around the starting pixel is hopelessly wrong — the gradient there says nothing about where the ball landed. The fix is a coarse-to-fine image pyramid. Shrink both frames repeatedly (each level halves the resolution); on a 16×-smaller image that 40-pixel jump becomes a gentle 2.5-pixel nudge, well within the small-motion regime, so we estimate flow reliably there. Then we upscale that flow estimate, use it to pre-warp the next finer level (cancelling most of the motion), and solve only for the small leftover — refining level by level up to full resolution. Big motions become solvable by always working where motion looks small.

Modern deep flow and how flow gets used

Everything so far was hand-designed: we wrote down brightness constancy and a smoothness assumption and solved the resulting equations. Modern dense flow keeps exactly the same two intuitions — match brightness, stay smooth — but learns them from data with a neural network trained end-to-end. The pioneer, FlowNet, simply fed a pair of frames into a convolutional network and had it regress a full flow field, learning from large datasets of synthetic videos with known ground-truth motion. The data term and smoothness prior are no longer typed in by hand; they are absorbed into the network's learned weights, which is why deep flow handles lighting changes, blur, and texture-poor regions far more gracefully than the classical methods.

The dominant modern design, RAFT, is worth a one-paragraph sketch because it mirrors our story so cleanly. First it builds a correlation volume: for each pixel in frame 1 it computes how well that pixel's features match every candidate location in frame 2 — a brute-force, learned version of brightness matching that, crucially, can see large displacements directly (no pyramid required). Then, instead of predicting the flow in one shot, it iteratively refines an estimate: a recurrent update module repeatedly looks up the correlation volume at the current flow guess and nudges the flow a little closer, dozens of times. That iterative nudging is conceptually the same loop as coarse-to-fine refinement, learned rather than hand-coded. Conceptually nothing new — match, then smooth-and-refine — but executed by a trained network, it is dramatically more accurate.

To judge any of these methods we need a number, and the standard one is the Endpoint Error (EPE) — quite simply, how far the predicted arrow's tip lands from the true arrow's tip.

Average Endpoint Error: mean Euclidean distance between predicted and ground-truth flow vectors over all pixels.

Symbol by symbol: for one pixel p, (u_pred, v_pred) is the flow vector our method predicted and (u_gt, v_gt) is the ground-truth flow ('gt' = ground truth, the known correct answer). Their difference is itself a 2D vector — the error arrow — and ‖·‖₂ is its Euclidean length (the ordinary √(Δu² + Δv²) straight-line distance). We compute that length at every pixel and average over all N pixels (the 1/N sum). The intuition is wonderfully direct: EPE answers 'on average, by how many pixels does my predicted arrow miss the true arrow?' A concrete number: if at one pixel I predict (3, 4) but the truth is (0, 0), the error vector is (3, 4) with length √(9+16) = 5, so that pixel contributes an endpoint error of 5 pixels. Top methods on benchmark datasets reach average EPE well below 1 pixel.