optical flow

Optical flow is the apparent motion of brightness patterns between two consecutive video frames. For every pixel you produce a small arrow, a velocity vector written (u, v), that says how far and in what direction the content at that pixel seems to have shifted from this frame to the next. A vivid analogy: imagine wind blowing across a field of grass and you only get to see the grass, not the wind. The flow is your best guess about the wind from watching how the grass bends. Critically, flow is about how the picture's brightness moves, which is not always the same as how the physical world moves.

The workhorse assumption is brightness constancy: a point on a surface keeps the same intensity as it moves a tiny bit between frames. Writing I(x, y, t) for the image intensity at pixel location (x, y) at time t, this says I(x, y, t) = I(x + u, y + v, t + 1). A first-order Taylor expansion of the right side turns this into the optical flow constraint equation: I_x·u + I_y·v + I_t = 0, where I_x and I_y are the spatial gradients (how fast brightness changes going right and going down) and I_t is the temporal gradient (how fast brightness changes from frame to frame at a fixed pixel). This is one linear equation in two unknowns (u and v), so a single pixel can never pin down the full motion by itself.

That under-determination has a name: the aperture problem. Looking at a moving straight edge through a small hole, you can only measure the component of motion perpendicular to the edge (the normal flow); motion sliding along the edge is invisible. To recover full 2D flow you must add a constraint, either that nearby pixels move similarly (Lucas-Kanade, using a local window) or that the flow field is globally smooth (Horn-Schunck, a variational penalty). This is also why flow is well-determined at corners and textured regions but ambiguous along edges and impossible in flat, textureless areas.