multi-view stereo

Multi-view stereo is what produces the dense, detailed 3D you imagine when you think of photogrammetry: not a few thousand scattered feature points, but a depth value at essentially every pixel, fused across many photos into a complete surface. It picks up where structure-from-motion leaves off. SfM gives you the cameras' calibrated poses and a sparse point cloud; MVS uses those known cameras to reconstruct the geometry everywhere in between.

Its governing principle is photo-consistency. Consider a candidate 3D point on a surface. If you project it into every camera that can see it, those pixels should look alike — same color, same local texture — because they are all images of the same physical patch. A point at the wrong depth projects to mismatched, inconsistent pixels. So MVS hypothesizes depths and keeps the ones whose multi-view projections agree, typically scoring agreement with normalized cross-correlation or a learned similarity that tolerates lighting and viewpoint changes. This turns reconstruction into a per-pixel depth search regularized for smoothness and visibility.

Classical MVS comes in a few flavors. Plane-sweep stereo slices the scene into depth planes and tests photo-consistency at each. PatchMatch MVS, the core of COLMAP's dense stage, cleverly propagates good depth-and-normal guesses between neighboring pixels and refines them, jointly estimating local surface orientation for sharper results. The per-image depth maps are then fused — cross-checked for consistency across views and merged — into a single dense point cloud, which is finally turned into a watertight mesh by surface reconstruction (e.g. Poisson) and textured. Learned MVS (MVSNet and its many successors) replaces the hand-crafted cost with a neural cost volume built by warping image features onto depth hypotheses, often improving completeness in weakly textured regions.

MVS inherits stereo's hard cases and adds its own. Textureless surfaces give no photo-consistency signal; specular and transparent materials violate the brightness-constancy assumption outright; thin structures and concavities are hard to see from enough angles. Its great advantage over two-view stereo is redundancy: with many overlapping views, occlusions in some images are covered by others, and outliers are outvoted, so coverage and robustness improve markedly as you add well-distributed cameras.