The Camera in Matrices: From World to Pixel

Homogeneous Coordinates: Making Projection Linear

Remember the small headache from the previous guide on the pinhole model. To project a 3-D point onto the image, we had to divide by depth: a point at world depth Z lands at screen position proportional to X/Z and Y/Z. That division is what makes far things look small — but it is also what makes perspective projection awkward. A plain matrix can only add and scale its inputs; it can never divide one coordinate by another. So projection, as written in guide 4, simply cannot be one matrix multiply. This guide fixes that, and the fix unlocks the entire 'camera as a chain of matrices' story.

The trick is surprisingly cheap. We glue one extra number — a 1 — onto the end of every point. A 2-D pixel (x, y) becomes the triple (x, y, 1); a 3-D world point (X, Y, Z) becomes the four-tuple (X, Y, Z, 1). These padded vectors are called homogeneous coordinates. The whole point is to defer the division: everything in the middle — rotating, translating, applying focal length — becomes clean matrix multiplication, and the single annoying divide is postponed to one final clean-up step right at the end.

Lifting a point into homogeneous form (left) and projecting it back to the real plane (right).

Read it left to right. Going up: we take an ordinary point (x, y) and append the homogeneous scale 1, giving (x, y, 1). Going back down: a homogeneous vector [u, v, w] returns to the ordinary plane by dividing its first two entries by the last entry w, giving (u/w, v/w). That last w is exactly where the perspective divide-by-Z gets hidden. While we chain matrices, depth quietly accumulates in w; only at the very end do we divide by it. Concretely, if a chain spits out [480, 320, 2], the real pixel is (480/2, 320/2) = (240, 160).

Scale invariance: all of these homogeneous triples name the very same 2-D point.

Here the squiggle ~ means 'represents the same point'. Two homogeneous vectors are considered equal whenever one is a nonzero scalar multiple of the other. Why? Because dividing back collapses the scale away: [2x, 2y, 2] divides to (2x/2, 2y/2) = (x, y), the same as [x, y, 1]. So the 1 we appended was never special — any nonzero number would do, since the final divide normalises it out. This freedom is what buys us linearity: a matrix is free to rescale the whole vector (multiply everything by some k), because rescaling does not change which point we mean. There is even a bonus. Set the last coordinate to w = 0 and the division (u/0) blows up to infinity — so [u, v, 0] cleanly represents a point at infinity, the direction in which parallel rails meet. Those are exactly the vanishing points from guide 4, now first-class citizens we can compute with.

Camera Intrinsics: Inside the Camera

In guide 4 the pinhole model gave us a projected ray in physical units — think of it as 'how many millimetres up and across on the sensor'. But an image is not measured in millimetres; it is measured in pixels, counted from the top-left corner (recall guide 1). The job of converting a clean physical ray into an honest pixel address belongs to a single 3×3 matrix called the camera intrinsics, written K. It encodes everything about the lens-and-sensor unit itself, and for a given camera it never changes from shot to shot.

The intrinsic matrix K — five numbers that describe one camera's lens and sensor.

Let us name every entry. f_x and f_y are the focal length expressed in pixels — horizontally and vertically. c_x and c_y are the principal point: the pixel where the optical axis pierces the sensor. s is skew, which measures whether the pixel grid's axes are perfectly perpendicular. The bottom row [0, 0, 1] is pure bookkeeping: it keeps the output homogeneous (a valid [u, v, w] triple) so we can keep chaining. The pattern of the top two rows is exactly 'multiply by focal length, then add the centre offset' — the linear recipe we needed.

Two questions naturally arise. First, why two focal lengths f_x and f_y when guide 4 had a single f? Because a sensor pixel need not be square. The physical focal length f is one number in millimetres, but converting it to pixels means dividing by the pixel width horizontally and by the pixel height vertically; if those differ, you get two different pixel-focal-lengths. So f_x and f_y are not a new mystery — they are the same focal length f from guide 4, just measured along each axis in that axis's pixel size. On most modern phones pixels are square and f_x ≈ f_y. Second, why is the principal point not at (0, 0)? Because guide 1 put the image origin at the top-left, while the optical axis hits roughly the centre of the sensor. So (c_x, c_y) ≈ (width/2, height/2) re-centres the coordinates onto the pixel grid — for a 640×480 image, about (320, 240).

Skew s deserves one honest sentence: on essentially every real camera it is ~0, because sensor rows and columns are manufactured perpendicular. It only appears in exotic or deliberately sheared setups, so you can mentally cross it out and read K as four meaningful numbers (f_x, f_y, c_x, c_y) plus the bookkeeping row.

How does K actually land a ray on a pixel? Feed it the projected ray in homogeneous coordinates. Suppose the camera-frame point is (X_c, Y_c, Z_c). Multiplying K by [X_c, Y_c, Z_c] gives the homogeneous pixel [f_x·X_c + c_x·Z_c, f_y·Y_c + c_y·Z_c, Z_c]. Notice the third entry is exactly the depth Z_c — that is the w that will perform the perspective divide. Divide by it and you get u = f_x·(X_c/Z_c) + c_x and v = f_y·(Y_c/Z_c) + c_y: the guide-4 normalised ray X_c/Z_c, scaled by focal length and shifted to the image centre. K is literally 'focal length × ray, then recentre'.

Camera Extrinsics: Where the Camera Stands

Intrinsics describe the camera's insides; they say nothing about where the camera is or which way it looks. That second job belongs to the camera extrinsics — a rotation R and a translation t that together describe the camera's pose: its orientation and its position in the world. If intrinsics are fixed properties of the lens-and-sensor unit, extrinsics are the part that changes with every single shot, because every shot is taken from somewhere, pointing somewhere.

Here is the analogy to hold onto. Your eyes are a fixed optical instrument — that is the intrinsics, and they do not change when you walk around. But when you step across the room and turn your head, what is 'in front of you' changes completely. That stepping (translation) and turning (rotation) is exactly the extrinsics. The camera's job before projecting is to re-describe every world point from its own point of view — to transform the point from world coordinates into camera coordinates, the frame where the camera sits at the origin looking down its own axis. Only in that frame can the pinhole projection of guide 4 be applied.

The world-to-camera transform: rotate, then shift.

Symbol by symbol: X_world is the 3-D point written in the world frame (the shared, fixed coordinate system everyone agrees on). R is a 3×3 rotation matrix; multiplying by R re-orients the world's axes so they line up with the camera's viewing direction — it answers 'which way am I facing?'. t is a 3-vector translation that shifts the origin from the world's chosen zero to the camera's optical centre — it answers 'where am I standing?'. The result X_cam is that exact same physical point, now expressed from the camera's own viewpoint. Order matters: we rotate first, then add t. As a tiny check, if the camera sits at the world origin and is perfectly aligned (R is the identity, t is zero), then X_cam = X_world — the camera and world agree, exactly as you would hope.

The same transform in homogeneous form: R and t pack into one block.

This is the homogeneous payoff again. The separate 'multiply by R, then add t' becomes a single matrix multiply once we pad X_world with a 1. The 3×3 R and the 3-vector t sit side by side to form a 3×4 block written [R | t], with a bottom row [0, 0, 0, 1] keeping things square and homogeneous. Multiplying it by [X_world, 1] reproduces R·X_world + t exactly — the addition of t is now 'free', performed by the matrix. That 3×4 block [R | t] is precisely the piece the next section snaps together with K.

The Full Projection Pipeline: World → Pixel

Now we assemble the whole machine. Everything this track taught — a pixel grid, color, sampling, the pinhole geometry, and the three matrix ideas of this guide — collapses into one elegant equation that carries any 3-D world point straight to a pixel. There are exactly three stages, and they run in a fixed order: extrinsics place the point in the camera's frame, intrinsics turn that ray into pixels, and the final homogeneous divide performs the perspective shrink.

The full pinhole projection equation — the climax of the whole track.

Walk it right to left, tracing each symbol back to where the ladder introduced it. [X, Y, Z, 1] is the world point in homogeneous form (Section 1, with its appended 1). [R | t] is the extrinsics from Section 3; it maps the point from world coordinates into camera coordinates — this is the guide-4 geometry of 'put the scene in front of the lens'. K is the intrinsics from Section 2; it applies focal length (guide-4's f, split into f_x, f_y) and re-centres onto the pixel grid (guide-1's top-left origin). The right side is now a homogeneous triple. s is the scale that pops out as its third entry — and that s is nothing other than the camera-frame depth, the divide-by-Z of perspective projection from guide 4. Dividing the triple by s gives the honest pixel [u, v, 1]. One equation; the whole track inside it.

1. Lift the world point to homogeneous form: (X, Y, Z) → [X, Y, Z, 1].
2. Apply extrinsics [R | t] to get the point in camera coordinates (where it sits relative to the lens).
3. Apply intrinsics K to scale by focal length and shift to the image centre — output is a homogeneous pixel [u', v', s].
4. Read off the scale s (the depth), then divide: the final pixel is (u'/s, v'/s).

Let us run one concrete point through the entire machine. Take a 640×480 camera with K given by f_x = f_y = 800, c_x = 320, c_y = 240, s = 0. Let the camera be aligned with the world axes so R is the identity, and let it be backed off by t = (0, 0, 6) — six metres behind the world origin along its viewing axis. We will project the world point (X, Y, Z) = (2, 1, 4).

Extrinsics then intrinsics, leaving the depth as the scale s = 10.

Stage by stage: extrinsics turn the world point (2, 1, 4) into the camera-frame point (2, 1, 10) — the depth grew from 4 to 10 because the camera sits 6 m back. Then K multiplies: the first row gives 800·2 + 320·10 = 1600 + 3200 = 4800, the second row 800·1 + 240·10 = 800 + 2400 = 3200, and the third row simply carries the depth, 10. So the homogeneous pixel is [4800, 3200, 10] and the scale is s = 10 — exactly the camera-frame depth, confirming s really is the perspective divide. The final step divides: u = 4800/10 = 480 and v = 3200/10 = 320. The point lands at pixel (480, 320) — inside our 640×480 image, just right and below centre. The whole machine ran end to end and produced clean integer pixels.

import numpy as np

# Intrinsics: focal length in pixels, principal point at image center
K = np.array([[800,   0, 320],
              [  0, 800, 240],
              [  0,   0,   1]])

# Extrinsics: camera aligned with world (R = I), backed off 6 m along Z
R = np.eye(3)
t = np.array([0, 0, 6])
Rt = np.hstack([R, t.reshape(3, 1)])   # 3x4 block [R | t]

# A world point in homogeneous coordinates
Xw = np.array([2, 1, 4, 1])

# Full pipeline: extrinsics, then intrinsics
hom_pixel = K @ Rt @ Xw                  # = [4800, 3200, 10]
s = hom_pixel[2]                         # the scale = camera-frame depth
u, v = hom_pixel[0] / s, hom_pixel[1] / s
print(u, v)                             # -> 480.0 320.0

Real Lenses Break the Pinhole: Distortion

Our elegant equation assumes a true pinhole. But recall the brightness trade-off from guide 4: a pinhole small enough to be sharp lets in almost no light, so real cameras replace the hole with a lens that gathers far more light and focuses it. The catch is that no real lens bends every ray perfectly. The deviation between where a ray actually lands and where the ideal pinhole says it should land is called lens distortion, and it is the single biggest gap between our tidy model and a real photograph.

The dominant kind is radial distortion: the error grows with distance from the image centre, so it warps straight lines into curves. Two flavours appear. In barrel distortion, lines bow outward and the image looks like it is wrapped around a barrel — exactly the look of a GoPro or fisheye, common in wide-angle lenses with a large field of view. In pincushion distortion, lines bow inward, common at the telephoto end. A vivid test: photograph a straight wall edge near the frame border with a wide-angle phone lens and watch it curve, even though you know the wall is dead straight.

The radial distortion model: a radius-dependent stretch applied to the ideal coordinates.

Symbol by symbol: (x, y) are the ideal normalised image coordinates — the clean pinhole positions, measured relative to the centre. r is the distance of that point from the image centre, so r² = x² + y². k_1, k_2 are the radial distortion coefficients — a couple of small numbers that characterise this particular lens (k_1 < 0 gives barrel, k_1 > 0 gives pincushion). The factor (1 + k_1 r² + k_2 r⁴ + …) is a stretch that depends only on r. (x_d, y_d) are the resulting distorted coordinates where the ray truly lands. The crucial intuition: at the centre r = 0, the factor is exactly 1, so nothing moves — the middle of the image stays clean. As r grows toward the corners, the r² and r⁴ terms swell, so the corners are pushed or pulled the most. That is precisely why straight lines look fine through the middle but bow near the edges. Tiny example: with k_1 = 0.1, a point at r = 1 is stretched by factor 1.1 (a 10% push), while a point at r = 0.5 (r² = 0.25) is stretched by only 1.025 — four times less.

There is a second, smaller effect: tangential distortion, which arises when the lens and sensor are not perfectly parallel (a slight manufacturing misalignment). It shifts points sideways rather than radially and is modelled by two more coefficients, usually written p_1 and p_2. Together the radial k's and tangential p's form the camera's distortion profile. The key mental model: distortion is an additive correction layer wrapped around the clean pinhole pipeline of Section 4 — the matrices K and [R | t] still do their job, and the distortion terms describe how reality nudges the ideal pixel off its predicted spot.

Calibration and Why All This Matters

We have built a model with unknowns inside it: the intrinsics K, the distortion coefficients, and a set of extrinsics for each shot. Camera calibration is the procedure that fills in those numbers. The idea is beautifully simple: show the camera an object whose geometry you already know exactly, photograph it from several angles, and let the software work backward to whatever camera parameters would make the model's predictions match what the photos actually recorded.

The classic known object is a checkerboard. Its corners sit on a perfect grid with known spacing, so for each photo the software can detect where every corner appears in pixels and compare that to where the model predicts it should appear. The mismatch is an error to minimise. The flow is 'known geometry in → unknown camera parameters out': you feed in something you fully understand (the board) so the only thing left to explain the images is the camera itself.

1. Print a checkerboard of known square size and tape it to a flat board.
2. Take 10–20 photos of it from varied angles and distances, filling different parts of the frame.
3. Detect every inner corner in each image, in pixel coordinates.
4. Solve for the K, the distortion coefficients, and a per-shot [R | t] that best reproduce all the detected corners.
5. Use the recovered K and distortion to undistort future images and project accurately.

Why go to all this trouble? Because a trustworthy camera model is the foundation for everything that has to undo projection — the hard inverse problem flagged back in guide 4. Forward projection throws depth away (many 3-D points collapse to one pixel); recovering 3-D structure from images only works if you know precisely how the camera mapped world to pixel. Calibration is what makes that knowledge precise. With it you can do 3-D reconstruction, measure real-world distances from photos, compute stereo depth from two cameras, plant convincing virtual objects into a scene for augmented reality, and let robots and self-driving cars judge where things actually are.

That closes this track: you can now follow a point of light from the world, through the lens and sensor, into a grid of pixels — and write the whole journey as one chain of matrices. What comes next builds directly on this foundation. The multi-view geometry tracks ahead use two or more calibrated cameras to triangulate depth and reconstruct full 3-D scenes — turning the projection you have just mastered around, and running it backward.