lucas-kanade method

The Lucas-Kanade method resolves the aperture problem with one clean assumption: all pixels inside a small window move together with the same flow (u, v). A single pixel gives you only one equation (the optical flow constraint), but if you trust that, say, a 15×15 patch shares one motion, you now have 225 equations for the same two unknowns. Two unknowns, many equations: that is an over-determined linear system, and the natural answer is the least-squares solution that makes all the constraints as satisfied as possible at once.

Concretely, stack the per-pixel equations I_x·u + I_y·v = -I_t into a matrix system A·v = b, where each row holds a pixel's gradients (I_x, I_y), v = (u, v) is the unknown flow, and b holds the negative temporal gradients. The least-squares solution comes from the normal equations (AᵀA)·v = Aᵀb. The 2×2 matrix AᵀA is the structure tensor: it sums I_x², I_xI_y, and I_y² over the window. The flow is reliable exactly when this matrix is well-conditioned, meaning both its eigenvalues are large, which happens at corners. On an edge it is rank-deficient (one eigenvalue near zero), so the aperture problem returns as an unsolvable direction. This is the same matrix that drives the Harris corner detector, which is why good features to track are corners.

Two refinements make it practical. First, the brightness-constancy Taylor expansion is only valid for small motion, so large displacements are handled with coarse-to-fine image pyramids: estimate flow on a heavily downsampled image where motion is tiny, then warp and refine level by level up to full resolution. Second, within each level the solution is iterated (a Gauss-Newton update): warp the second frame by the current estimate, recompute the residual I_t, and solve again until convergence. Tracking a sparse set of corner features across many frames this way is the classic KLT tracker (Kanade-Lucas-Tomasi).