From Pixels to Geometry: How a Camera Sees

Why 3D Vision? The Problem of Lost Depth

Point a camera at the world and something quietly dramatic happens: a rich three-dimensional scene — with near things and far things — gets squashed onto a flat sheet of pixels. The sheet faithfully records where light landed and how bright it was, but it forgets one number for every point: how far away that point actually was. That missing number is depth, and recovering it is the obsession of this entire track.

Think of a shadow puppet. Your hand is a real 3D object, but the lamp throws only its outline onto the wall. From that flat silhouette alone you genuinely cannot tell whether the hand is small and close to the wall or large and far from it — many different hands cast the very same shadow. A camera performs exactly this kind of flattening, and it leads to the single most important fact in this guide: infinitely many 3D points map to the same pixel. Every point lying along one straight line of sight (a ray) out of the camera lands on the identical spot of the image.

How do you invert something that genuinely lost information? By putting the information back. This track climbs a ladder of strategies: start with one camera and understand exactly how it projects (this guide); add a second camera so two views can triangulate depth, the way your two eyes do; chain many cameras / many frames to reconstruct whole scenes and track the camera's path; and finally let neural networks learn depth from data when geometry alone runs out. By the end of the track, flat photos become navigable 3D.

The Pinhole Camera Model

A real camera obscura ('dark room') is the ancestor of every camera. Poke a tiny hole in one wall of a dark box. Light from a point in the scene travels in a straight line, passes through the hole, and strikes the opposite inside wall. Because each scene point sends its ray through that one shared hole, the picture forming on the back wall is a perfect — though upside-down — image of the world. Let us name the parts. The hole is the optical centre (or pinhole); the line straight out of the hole, perpendicular to the back wall, is the principal axis; the back wall where the image forms is the image plane; and the distance from the pinhole to the image plane is the focal length, written f.

Now the geometry. Set up coordinates at the pinhole: let Z measure straight ahead along the principal axis (this is depth), and let X and Y measure sideways and up. Take a scene point at (X, Y, Z). The ray from that point to the pinhole, and the ray continuing on to the image plane, form two similar triangles that meet tip-to-tip at the pinhole. One triangle has 'height' Y and 'base' Z (point-to-pinhole); the other has height v (where it lands on the image) and base f (pinhole-to-image-plane). Similar triangles have equal ratios of corresponding sides — and that single fact is the entire engine of perspective.

Perspective projection: equal side-ratios of the two similar triangles, rearranged.

Read it out loud. (X, Y, Z) is the 3D point in the camera's own coordinate frame, with Z its depth — how far straight ahead it sits. f is the focal length, in the same length units as X, Y and Z. (u, v) is the spot the point lands on the image plane. The equation says: take the sideways position X, scale it by f, then divide by the depth Z. That division is everything. Put a 1-metre-wide object at Z = 2 m and it images at size f/2; push it to Z = 4 m — twice as far — and it images at f/4, exactly half the size. Double the distance, halve the image. This is why railway tracks shrink toward a point and distant friends look tiny — not an optical quirk, just dividing by Z.

Homogeneous Coordinates: The Mathematician's Trick

The projection u = fX/Z is correct but mathematically awkward. Almost everything else in vision — rotating the camera, moving it, chaining several cameras — is done with matrices, because matrices compose cleanly (do one step, then the next, just multiply them together). But our projection has a division by Z buried inside it, and division is not something a single matrix multiplication can perform. We need a way to turn 'divide by Z' into the language of matrices. That way is homogeneous coordinates, and every later guide leans on it.

The idea is almost cheeky: add one extra coordinate. Represent a 2D image point (u, v) by the triple (u, v, 1), and a 3D point (X, Y, Z) by (X, Y, Z, 1). Then adopt one new rule — scale does not matter: a triple (x, y, w) stands for the ordinary point you get by dividing through by the last entry, (x/w, y/w). So (4, 6, 2), (8, 12, 4) and (2, 3, 1) are all the same image point (2, 3). Geometrically you have lifted the plane up by one dimension, and every line through the origin in that bigger space now represents a single point of the plane. The pay-off: projection becomes one tidy matrix multiply, and the annoying division is deferred to a single final step — 'divide by the last coordinate' — done just once at the very end.

A homogeneous image point is defined only up to a nonzero scale s.

Symbol by symbol: the tilde ~ means 'equal up to scale' — both sides denote the same image point even though their numbers differ. s is any nonzero scale factor you like; you are free to multiply a homogeneous vector by any s ≠ 0 without changing what it represents. To recover real pixels you normalise: divide the first two entries by the third, (su, sv, s) → (u, v). And this is where all the perspective division from Section 2 hides: a matrix will produce (su, sv, s) in which s turns out to equal the depth Z, so the final divide-by-s is the divide-by-Z. The mess never vanished — we simply postponed it into one clean, predictable step.

# A homogeneous image point, and how to read off real pixels.
# Equivalence "up to scale": all three of these are the SAME pixel.
p1 = [4.0, 6.0, 2.0]    # (s*u, s*v, s) with s = 2
p2 = [8.0, 12.0, 4.0]   # the same point, scaled again by 2
p3 = [2.0, 3.0, 1.0]    # the canonical form (last entry = 1)

def to_pixel(h):
    s = h[2]                       # the last (scale) coordinate
    return (h[0] / s, h[1] / s)    # divide it out -> (u, v)

to_pixel(p1)   # -> (2.0, 3.0)
to_pixel(p2)   # -> (2.0, 3.0)   same pixel!
to_pixel(p3)   # -> (2.0, 3.0)

Intrinsics & Extrinsics: The Full Camera Matrix

We can now assemble the complete camera in one expression, by separating two jobs a camera secretly does at once. Job 1 — where is the camera? The world has its own coordinate frame (say, a corner of the room as origin), yet our projection formula assumed the point was already given in the camera's frame. So first we must carry world points into camera coordinates. Job 2 — how does this particular lens-and-sensor turn metres on the image plane into pixels? Job 1 is the extrinsics; Job 2 is the intrinsics.

Extrinsics answer 'where the camera is and which way it points', and they take the form [R | t]. R is a 3×3 rotation that twists the world's axes to line up with the camera's (which direction it faces); t is a 3-vector translation that shifts the origin (where it sits). Glued side by side they form a 3×4 block: feed it a world point and out comes that same point expressed in camera coordinates, ready for projection. The analogy: extrinsics are posing the tripod — carry the camera to a spot and aim it. Move the camera and only [R | t] changes.

Intrinsics, the matrix K, are the camera's fixed inner personality — they do not change as you walk around. K converts a projected point on the image plane into actual pixel row/column numbers. It carries four meaningful values: f_x and f_y are the focal length measured in pixels (two of them, because sensor pixels need not be perfectly square); (c_x, c_y) is the principal point, the pixel where the principal axis pierces the sensor — usually near the image centre; and a skew term that is essentially always 0 for real cameras. The analogy: intrinsics are the lens-and-sensor's personality, baked in at the factory.

The full projection: place the point, then pixel-ise it.

Here is the entire camera on one line. X is a homogeneous 3D world point [X, Y, Z, 1]ᵀ. [R | t] is the 3×4 extrinsic block that carries it into camera coordinates. K is the 3×3 intrinsic matrix that turns that into pixels. The result x = [su, sv, s]ᵀ is a homogeneous pixel, so you finish by dividing out s to read off the actual column u and row v — the very normalise step from the last section. The ~ reminds you the equality holds only up to scale. Read right to left, it is just our whole story stacked: place the point (extrinsics), project and pixel-ise (intrinsics), then normalise.

The intrinsic matrix, entry by entry (skew s ≈ 0 in practice).

Spelled out, K is upper-triangular: top row (f_x, s, c_x), middle row (0, f_y, c_y), bottom row (0, 0, 1). That bottom-row 1 is exactly what makes the output's last coordinate carry the depth-scale for normalising. Multiply K by a camera-frame point (X_c, Y_c, Z_c)ᵀ and you get (f_x·X_c + c_x·Z_c, f_y·Y_c + c_y·Z_c, Z_c)ᵀ; divide by that last Z_c and you land back on u = f_x·X_c/Z_c + c_x — Section 2's formula, now in pixels with the principal-point offset added on. This one equation x ~ K[R|t]X is the equation every 3D-vision algorithm secretly inverts, which is exactly why we must first measure K. Getting K is precisely camera calibration — the topic of the next section.

Camera Calibration: Measuring Your Camera

Those numbers f_x, f_y, c_x, c_y are not printed on the side of your camera, and they differ from lens to lens (and shift when you zoom). Until you know them, every pixel-to-geometry calculation is guesswork, and any 'depth in metres' you later compute is meaningless. The procedure that measures them — plus the lens's distortion — is camera calibration. The classic, wonderfully practical recipe is Zhang's method: the checkerboard you have surely seen waved in front of a webcam.

1. Print a checkerboard with squares of a known size and fix it to something flat. Because the pattern is regular and rigid, you know the real-world (X, Y) of every corner exactly — and you may set its Z to 0, since the board is a single plane.
2. Photograph the board from roughly 10–20 different angles and distances, tilting it so every shot looks geometrically different.
3. In each photo, automatically detect the checkerboard corners to sub-pixel accuracy — these are the (u, v) pixel positions.
4. You now hold many known 3D-corner ↔ 2D-pixel pairs. Solve backwards for the unknowns that best explain them all at once: the single shared K (and distortion), plus a separate [R | t] for each photo.
5. Optimise (a least-squares fit) to minimise the total gap between where corners actually landed and where K[R|t] predicts they should — then keep K and the distortion coefficients.

import cv2, numpy as np

# 3D corner positions on the flat board, in board coordinates (Z = 0).
# Identical for every photo, because the board itself never changes.
objp = np.zeros((6 * 9, 3), np.float32)
objp[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2)

obj_points, img_points = [], []        # 3D<->2D pairs, one set per photo
for img in checkerboard_photos:        # ~15 views from many angles
    found, corners = cv2.findChessboardCorners(img, (9, 6))
    if found:
        obj_points.append(objp)        # where the corners ARE (world)
        img_points.append(corners)     # where they LAND  (pixels)

# Solve backward for the camera: K, distortion, and each view's R, t.
err, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, image_size, None, None)

Real lenses bend light a little more than the perfect pinhole predicts, especially out toward the edges of the frame. The visible symptom is radial distortion: straight lines near the border bow outward like a barrel (or pinch inward like a pincushion). Calibration also estimates a few distortion coefficients so software can undistort the image — straighten those lines back — before any geometry is trusted. The most common model corrects each point by how far it sits from the centre:

The radial distortion model (shown for the x-coordinate; y is identical).

Term by term: x is the ideal (undistorted) point in normalised image coordinates, measured from the principal point; r is simply its distance from the centre, r = √(x² + y²); and k₁, k₂ are the radial distortion coefficients found during calibration. The bracket (1 + k₁r² + k₂r⁴) is a gentle stretch factor. Notice it depends only on r: at the very centre r = 0, so the factor is exactly 1 (no distortion there), but toward the corners r grows and — crucially — it enters as r² and r⁴, so the correction climbs fast. That is precisely why the edges warp far more than the middle. A positive k₁ pushes points outward (barrel distortion); a negative k₁ pulls them inward (pincushion). With K and these coefficients in hand, a calibrated camera finally yields metric, trustworthy geometry.

One Image Isn't Enough: The Scale Ambiguity

Here is the humbling punchline. Even with a flawlessly calibrated camera — K and distortion known to the decimal — a single image still cannot tell you metric depth. Recall why: projection multiplies X by f/Z, so a toy car 30 cm away and a real car 6 m away, sized to match, fall on exactly the same pixels. The depth Z divided out, and it carried the absolute scale of the world out with it.

Photographers exploit this every day. The tourist who appears to 'hold up' the Leaning Tower of Pisa, or to pinch the sun between two fingers, is using forced perspective: a small near thing and a huge far thing line up along the same rays, so the flat photo cannot separate them. This is no trick of a special lens — it is the projection equation behaving exactly as we derived it. We name this fundamental limit the scale (or depth) ambiguity: from one view, the world is known only up to an unknown overall scale.

There are exactly two ways to put the missing information back. Route 1 — geometry: add another viewpoint. Look from a second position and each scene point now sits on two rays; where those rays cross fixes its depth by triangulation, which is how your two eyes judge distance. Route 2 — priors: learn from data. Train a network on millions of images so it guesses depth from familiar cues (a car is roughly car-sized; textures shrink with distance) — this is monocular depth estimation, a learned form of depth estimation from a single image. Geometry is exact but needs more views; learning needs only one image but inherits whatever its training data quietly assumed.

We will walk both routes. Up next is the geometric one at its simplest and most beautiful — two cameras placed side by side, just like your eyes, turning a pair of flat pictures into a genuine map of depth. On to stereo vision.