bundle adjustment

Bundle adjustment is the final, all-at-once polishing step that makes structure-from-motion and visual SLAM accurate. After you have rough estimates of every camera's pose and every 3D point, bundle adjustment nudges all of them simultaneously to make the reconstruction maximally self-consistent. The name comes from the bundles of light rays converging at each camera center: you are adjusting all those bundles together so they best agree with what the cameras actually photographed.

The quantity it minimizes is the total reprojection error. For every observation — every time a particular 3D point was seen in a particular image — you take the current 3D point, project it through the current camera model, and measure the pixel distance between that prediction and the actually detected feature. Bundle adjustment finds the camera parameters and point coordinates that minimize the sum of all these squared pixel errors at once. It is a single, very large nonlinear least-squares problem; the variables are all the camera poses (and optionally intrinsics) plus all the 3D points, often numbering in the millions.

Solving it naively would be hopeless, but the problem has a saving structure: each 3D point is usually seen by only a few cameras, so the giant system of equations is extremely sparse. The standard solver, Levenberg-Marquardt, exploits this with the Schur complement trick: it eliminates the many point variables in a block step, leaving a much smaller dense system in just the camera variables to solve, then back-substitutes for the points. Libraries like Ceres Solver, g2o, and GTSAM implement this efficiently, and the structure is what lets bundle adjustment scale to city-sized reconstructions.

Two cautions matter. First, it is a local optimizer: it refines a good initial guess but can be trapped by a bad one, which is why incremental SfM interleaves it with careful initialization. Second, the squared-error objective is sensitive to outliers — a few wrong feature matches can drag the whole solution off — so practical bundle adjustment uses robust loss functions (Huber, Cauchy) that cap the influence of large residuals. Done right, bundle adjustment is what turns a drifting, locally-consistent guess into a globally accurate model.