Two Eyes, One Depth: Stereo Vision

How Two Eyes See Depth

Let's start with an experiment you can do right now. Hold one finger up at arm's length, and pick something far away behind it — a wall, a doorway, a tree. Now close your left eye and look with only the right; then switch and look with only the left. Your finger appears to jump sideways against the background. The background barely moves, but the finger leaps. That jump is the entire secret of stereo vision, and by the end of this guide you'll be able to turn it into a distance in metres.

Your two eyes sit a few centimetres apart, so each one captures a slightly different view of the world. Your brain fuses these two images into one and reads the amount of sideways shift as depth. The rule is beautifully simple: near things shift a lot, far things barely move at all. Something at infinity — the moon, a distant mountain — looks like it sits in exactly the same place in both eyes. That graded amount of shift, large for near and tiny for far, is the raw signal depth perception is built on.

Computer vision gives this shift a name: disparity. In plain words, disparity is how far a single scene point lands to the left in one image versus the other — measured in pixels. A point right under your nose has large disparity; a point on the horizon has almost zero. This whole field of stereo geometry is about measuring that pixel shift precisely and converting it into real distance. We collect every pixel's disparity into a disparity map, and from that map we will reconstruct the 3D world.

The Stereo Rig & Rectified Images

To turn this intuition into measurement, we build a stereo rig: two identical, calibrated cameras mounted side by side, pointing in the same direction, separated horizontally by a fixed distance called the baseline, written B. "Identical and calibrated" means we already know each camera's focal length and principal point from Guide 1's calibration step, and the two cameras share the same internal settings. "Same direction" means their optical axes are parallel — like two eyes both staring straight ahead rather than crossing. This neat arrangement is what makes the maths clean.

In reality no two cameras are mounted perfectly. The lenses may tilt slightly, the rows of pixels won't be exactly level with each other, and a point that lands on row 200 in the left image might land on row 205 in the right. So before matching, we apply a one-time warp to both images called rectification. Rectification re-projects the two images onto a common plane so that their rows become perfectly aligned: after it, any scene point appears on the same horizontal row in both the left and right image.

Why is this such a big deal? Without rectification, finding a pixel's partner means searching the whole other image — a 2D hunt over every (x, y). That is slow and error-prone. After rectification, the partner is guaranteed to be on the same row, so the search collapses to a 1D scan along a single horizontal line (called a scanline). We only have to slide left and right and ask "is it here? here? here?". Shrinking the search space from a 2D plane to a 1D line is what makes stereo matching fast enough to be practical and far less ambiguous.

Disparity to Depth: The Key Equation

Now the payoff. We want one formula that takes a disparity in pixels and hands us a depth in metres. We get it from similar triangles, using the same pinhole camera model from Guide 1. Picture two pinhole cameras, left and right, separated by baseline B, optical axes parallel. A scene point sits at depth Z in front of them. Because the two cameras view it from slightly different positions, the point projects to a slightly different horizontal position in each image — and the difference between those two positions is exactly the disparity d. Tracing the rays from the point through each pinhole down to the sensor forms two triangles with the same shape, and their proportions lock together into one clean relationship.

The fundamental stereo equation: depth from disparity.

Let's name every symbol. Z is the depth we want — how far the point is from the cameras, in metres. f is the focal length measured in pixels, which we obtained from the calibration in Guide 1 (it bundles the lens focal length and the sensor's pixel size into a single number). B is the baseline, the physical distance between the two cameras, in metres. d is the disparity, the horizontal pixel shift of the matched point between the left and right images. Read the formula as a recipe: multiply how 'zoomed in' the camera is (f) by how far apart the eyes are (B), then divide by how much the point jumped (d). A big jump means it's close; a tiny jump means it's far.

The single most important thing to feel here is the inverse relationship: depth is inversely proportional to disparity, Z ∝ 1/d. Double the disparity and the depth halves. As a point recedes into the distance, its disparity shrinks toward zero, and as d → 0, Z → infinity. This is the maths behind "the moon doesn't move between your eyes." It also carries a warning: because the relationship is 1/d, the same matching is precise up close and increasingly uncertain far away. Near the camera, depth changes gently with each pixel of disparity; far away, a single pixel of disparity covers an enormous range of depth.

Let's plug in real numbers. Suppose f = 800 pixels, B = 0.12 m (a 12 cm baseline, roughly the gap between human eyes), and we matched a point with disparity d = 40 pixels. Then Z = (800 × 0.12) / 40 = 96 / 40 = 2.4 m. Now take a far point with d = 4 pixels: Z = 96 / 4 = 24 m, ten times farther for one-tenth the disparity — exactly the inverse law in action. The units work because f·B has units of pixel·metre, and dividing by d (pixels) leaves metres.

Finding Correspondences: The Matching Problem

The depth equation assumes we already know the disparity d. But where does d come from? This is where the real work of stereo lives. To get d for a pixel in the left image, we must find that same physical point in the right image and measure how far it shifted. Doing this for every pixel is called the correspondence or matching problem, and it is the part that makes stereo genuinely hard. Thanks to rectification we don't search the whole image — only along the one scanline. But we still have to decide, among all the candidate positions on that line, which one is the true match.

The workhorse method is block matching. Instead of comparing one lonely pixel (a single brightness value is far too ambiguous), we grab a small window — say 5×5 or 9×9 pixels — around the left pixel, and slide that patch along the right scanline. At each candidate shift d we compare the left patch against the right patch and score how alike they look. The shift that makes the two patches look most similar wins, and that winning shift is our disparity. A window works far better than a single pixel because a little patch of texture is much more distinctive than one grey value.

How do we score 'similarity'? With a matching cost. Three classic choices: SSD (Sum of Squared Differences) adds up the squared brightness differences between the patches — squaring punishes big mismatches hard. SAD (Sum of Absolute Differences) adds up the plain absolute differences — cheaper and more forgiving of a single bad pixel. NCC (Normalised Cross-Correlation) measures the patches' correlation after removing their average brightness and scale, so it stays robust when one camera is simply brighter than the other. Low SSD/SAD means a good match; high NCC (near 1) means a good match.

Sum of Squared Differences as a function of the candidate disparity d.

Let's read this term by term. I_L(x, y) is the brightness (intensity) of the left image at pixel (x, y); I_R(x − d, y) is the brightness of the right image at the same row y but shifted left by the candidate disparity d. We subtract them, square the difference so positives and negatives can't cancel, and sum over W, the small window of pixels around the point. The result SSD(d) is one cost number for each candidate d; the best disparity is the d that minimises this cost. Notice that on the same row y — that is exactly the rectification payoff letting us shift only in x. Concretely, if a 9×9 window over a sharp corner gives SSD = 120000 at d = 39, SSD = 4000 at d = 40, and SSD = 90000 at d = 41, the sharp dip at 40 tells us the match is there.

def block_match_ssd(left, right, x, y, win=4, d_max=64):
    # Find the disparity d that best matches the left window
    # to a shifted right window, along the same scanline y.
    best_d, best_cost = 0, float('inf')
    for d in range(0, d_max):
        cost = 0.0
        # Sum squared brightness differences over a (2*win+1) square window
        for dy in range(-win, win + 1):
            for dx in range(-win, win + 1):
                il = left[y + dy][x + dx]
                ir = right[y + dy][x + dx - d]   # shift LEFT by d, same row
                diff = il - ir
                cost += diff * diff
        if cost < best_cost:      # keep the disparity with the lowest cost
            best_cost, best_d = cost, d
    return best_d                  # this is the disparity for pixel (x, y)

From Disparity Map to 3D Point Cloud

So far we've found the disparity for one pixel. Now scale up to the whole image: run block matching at every pixel and store each result in a new image where the value at each location is that pixel's disparity. This is the disparity map. We usually visualise it as a grayscale picture where bright means near (large disparity) and dark means far (small disparity). Glancing at a good disparity map, you can practically read the scene's 3D shape — a person pops out bright against a dark background, like a depth-coloured silhouette.

A disparity map is still a flat 2D image, though — every entry is just a number. The magic is that each of those numbers, fed through the depth equation, gives the metric depth Z at that pixel. And once we know Z, we can recover the full 3D position (X, Y, Z) of the point by back-projecting through the same pinhole model from Guide 1. Do this for every pixel and you produce a point cloud: a cloud of thousands of (X, Y, Z) points reconstructing the actual surfaces in front of the camera. This is the first time in the track you build real, dense 3D geometry from images.

Back-projecting a pixel to a metric 3D point, using the depth from disparity.

Read it term by term. (u, v) is the pixel's column and row in the image. (c_x, c_y) is the principal point — where the optical axis pierces the sensor, which we measured during Guide 1's calibration; subtracting it re-centres coordinates so the optical axis is the origin. f is the same focal length in pixels, and Z is the depth we just computed from the disparity d via Z = f·B/d. The result (X, Y, Z) is the point's real position in metres relative to the camera. The X and Y equations are nothing new — they are simply the pinhole projection of Guide 1 solved backward. Forward, the pinhole squashed (X, Y, Z) into a pixel and lost Z; now that the second view has handed us Z, we can run the projection in reverse and recover X and Y too.

There's a beautiful geometric picture behind these equations: this whole step is triangulation. The left pixel defines a ray shooting out from the left camera into space; the matched right pixel defines a second ray from the right camera. Two rays from two known viewpoints, aimed at the same physical point, intersect at exactly one place — and that intersection is the 3D point. The algebra above is just the bookkeeping that finds where the rays cross. Stack up the intersections for all pixels and you have your dense point cloud. The recipe in one line: disparity map + camera calibration = dense 3D via triangulation.

def disparity_to_point_cloud(disparity, f, B, cx, cy):
    # Turn a whole disparity map into a list of metric 3D points.
    points = []
    H = len(disparity)
    W = len(disparity[0])
    for v in range(H):            # v = row
        for u in range(W):        # u = column
            d = disparity[v][u]
            if d <= 0:            # 0 / unknown disparity -> infinitely far, skip
                continue
            Z = f * B / d         # depth from the key equation
            X = (u - cx) * Z / f  # back-project, same pinhole as Guide 1
            Y = (v - cy) * Z / f
            points.append((X, Y, Z))
    return points                  # the dense 3D point cloud, in metres

Limits of Stereo & Modern Twists

Let's close by stepping back. Stereo's strengths are genuinely valuable: it is cheap (two ordinary cameras and some computation, no exotic hardware) and it gives metric depth — actual metres, not a relative guess — straight from the geometry. That is why stereo rigs sit on robots, drones, cars, and AR headsets. But we've also been honest about the weaknesses: the matching failures on textureless walls, repeated patterns, and occlusions; and the inverse-law uncertainty that makes far-range depth shaky from even a single-pixel error. Stereo depth estimation is powerful but it is not magic, and knowing its failure modes is what separates a practitioner from a tinkerer.

The first upgrade practitioners reach for is Semi-Global Matching (SGM). Naive block matching decides each pixel in isolation, so noisy or ambiguous pixels guess wildly. SGM adds a smoothness assumption: neighbouring pixels probably belong to the same surface, so their disparities shouldn't jump for no reason. It penalises disparity changes between neighbours along many directions across the image and finds the disparity map that balances good matches against smoothness. The effect is dramatic — noisy speckle calms down and textureless regions borrow sensible values from their well-matched neighbours, while real edges are still allowed to jump.

Two more families dominate modern systems. Learned / deep stereo networks replace the hand-designed matching cost with a neural network trained on huge datasets; instead of comparing raw pixel patches, they compare learned features and even learn the smoothing, which lets them handle textureless and repetitive regions far more gracefully than SSD ever could. Active stereo / structured light attacks the root cause of failure — lack of texture — by projecting texture onto the scene: a projector throws a known pattern of dots or stripes (often in infrared) onto blank surfaces so even a featureless white wall suddenly has something to match. The original Microsoft Kinect worked exactly this way, which is why it could map a plain living-room wall that passive stereo would find hopeless.

One last thread to pull, leading into the next guide. Everything here leaned on a comfortable assumption: that our two cameras were neatly aligned and rectified, so matches lived on the same horizontal row. But the world rarely hands us such tidy rigs — what if the two views come from a single moving phone, or two photos taken from totally different angles? What geometry connects any two views of a scene, even un-aligned ones? That is the question the next guide answers, with the hidden rule we only glimpsed here: epipolar geometry. Master that, and you are ready to reconstruct the world from ordinary, messy photographs.