visual slam

Visual SLAM solves a chicken-and-egg problem in real time: a moving camera needs a map of its surroundings to know where it is, but it needs to know where it is to build that map. SLAM stands for Simultaneous Localization And Mapping, and 'visual' means the primary sensor is a camera. The result is a robot, drone, phone, or headset that, as it moves through an unknown space, continuously figures out its own pose (position and orientation) while incrementally constructing a map of the environment — all fast enough to act on, frame by frame.

A visual SLAM system has a few cooperating parts. The front end (tracking) processes each incoming frame to estimate the camera's motion relative to the recent map, by matching features or directly aligning pixel intensities. The back end (mapping) maintains and refines the map of 3D points and keyframes, typically with local bundle adjustment over a sliding window so it stays real-time. Crucially, a loop-closure module recognizes when the camera revisits a previously seen place; detecting that, the system corrects the accumulated drift across the whole trajectory with a global optimization (pose-graph optimization). Loop closure is what distinguishes SLAM from mere visual odometry, which tracks motion frame-to-frame but lets small errors accumulate without ever correcting the global map.

The methods split along how they use image information. Feature-based (indirect) SLAM, exemplified by ORB-SLAM and its successors, extracts and matches sparse keypoints (ORB features) and is robust and well-understood. Direct methods (LSD-SLAM, DSO) skip features and align images by raw photometric intensity, using more of the image and excelling in low-texture scenes. Systems also vary by sensor: monocular SLAM (one camera, suffers scale ambiguity), stereo SLAM (metric scale from a known baseline), and visual-inertial SLAM (VIO/VI-SLAM), which fuses an IMU to recover metric scale, handle fast motion, and stabilize tracking — the configuration behind most AR headsets and drones.

The frontier has moved toward dense and learned maps. Instead of sparse points, recent systems build dense surfaces or implicit representations: NeRF-based SLAM (iMAP, NICE-SLAM) and Gaussian-splatting SLAM optimize a photorealistic scene model online, and learned features and matchers (SuperPoint, SuperGlue/LightGlue) replace hand-crafted ones for robustness. The enduring challenges are the same as ever — fast motion and motion blur, low texture, dynamic objects that violate the static-world assumption, drift over long trajectories, and reliable relocalization after tracking is lost.