When Cameras Aren't Aligned
In the stereo guide we enjoyed a comfortable luxury: two cameras bolted side by side, looking the same direction, with their image planes perfectly parallel. That neat arrangement let a point in the left image slide straight across the same horizontal row to find its partner on the right. But the world is rarely so tidy. Take two phone snapshots of a statue from slightly different spots, or let a single camera drift across a scene while filming — now the two viewpoints sit at different positions and point in different directions. The horizontal-row trick collapses. We need a more general law.
Here is the question that organises this whole guide: given a single point in image 1, where can its match possibly be in image 2? It feels like the answer should be 'anywhere in image 2' — after all, we don't know the depth of that point yet. But that pessimism is wrong. Two photographs of one rigid scene are secretly bound together by geometry, and that binding shrinks the search dramatically. The study of that binding is called epipolar geometry.
The whole idea rests on one picture you should burn into memory. Pick a 3D point in the world, call it X. Light from X reached the first camera through its optical centre, which we call C. The same light reached the second camera through its centre C'. So we have three special points floating in space: the two camera centres C and C', and the world point X. Three points (when they aren't on a single straight line) always define exactly one flat plane. That plane is the hinge on which everything turns.
Two camera centres and a 3D point forming a triangle that lies in one plane; the point projects into each image, and a line is drawn across the second image.
The Epipolar Constraint
Let us name the cast of characters carefully, because the words come up constantly. First, the baseline: the straight line joining the two camera centres C and C'. It is the axis of the whole setup — think of it as the line between your two eyes, even when your eyes are tilted differently. Everything else hangs off the baseline.
Now extend the baseline until it pierces each image plane. Where it stabs through image 1 we get a special pixel called the epipole e; where it stabs through image 2 we get the epipole e'. Beautifully, the epipole in image 1 is literally the image of the other camera's centre — it is where camera 2 would appear if you could photograph it. (If the two cameras point roughly the same way, the epipoles may sit far outside the visible frame, even at infinity, which is fine.)
Finally, recall the epipolar plane through C, C' and X. Slice that plane against image plane 1 and you get a straight line — an epipolar line. Slice it against image plane 2 and you get the matching epipolar line there. And here is the headline result, the one fact this whole guide is built to deliver:
This is exactly the payoff we cherished in stereo, now in full generality: the search for a correspondence collapses from a 2D hunt across the whole image to a 1D walk along a single line. That is a massive saving, and it does double duty: it makes matching faster, and it lets us reject impossible matches. Any candidate pixel that does not sit on the epipolar line simply cannot be the true correspondence, no matter how similar it looks. We will lean on exactly this filter later when we clean up noisy matches. And once we do have a correct match on its line, intersecting the two rays back in 3D recovers the point's depth — the same triangulation idea from stereo, freed from the parallel-camera assumption.
The baseline joins both centres and meets each image at an epipole; the epipolar plane cuts each image in an epipolar line on which the match must lie.
The Fundamental Matrix
Geometry is lovely, but a computer wants algebra. Everything in the last section — the epipolar plane, the lines, the epipoles — can be bottled into a single 3×3 matrix called the fundamental matrix F. The very best thing about F is what it does not need: it works directly on raw pixel coordinates. You do not have to know the camera's focal length, sensor size, or any calibration to use it. That makes F the first tool you reach for when you grab two random photos off the internet and want to relate them.
F is special in a quiet but crucial way: it has rank 2, not the full rank 3 a generic 3×3 matrix would have. Rank 2 means its determinant is zero and it 'flattens' space by one dimension — and that missing dimension is exactly the epipoles. Concretely, the epipole e is the null vector of F (the direction F sends to zero, so that F·e = 0), and e' is the null vector of its transpose. So the rank-2 fact is not a technicality: it is the algebra remembering that two special points, the epipoles, exist. Any honest estimate of F must respect this, a point that will bite us in Section 5.
What does F actually do? It is a machine that turns a point in one image into the epipolar line in the other. Feed it the homogeneous pixel x from image 1 and it hands back the line l' = F·x in image 2 — and we already know the true match x' must sit on l'. That single sentence is the entire epipolar constraint, and we can write it as one tidy equation:
The epipolar constraint: the single scalar equation every true correspondence must satisfy.
Let us read it symbol by symbol. The vector x = [u, v, 1]ᵀ is a point in image 1 written in homogeneous coordinates — its pixel column u, row v, and a trailing 1 (the trailing 1 is the standard trick from Guide 1 that lets a 3×3 matrix handle both rotation-like and shift-like effects at once). The vector x' = [u', v', 1]ᵀ is the corresponding point in image 2. F is our 3×3 fundamental matrix. Now group the product as l' = F·x: this is a 3-vector encoding a line in image 2, written as coefficients [a, b, c] of the line a·u' + b·v' + c = 0. The remaining multiplication x'ᵀ·l' is just a dot product, and in homogeneous coordinates 'a point lies on a line' is exactly the statement that the point dotted with the line equals zero. So x'ᵀ F x = 0 literally says 'x' lies on the epipolar line of x'. The picture and the algebra are the same fact.
Concretely: suppose F·x comes out as l' = [1, 0, -200]ᵀ. That encodes the line 1·u' + 0·v' − 200 = 0, i.e. the vertical line u' = 200 in image 2. The constraint x'ᵀ l' = 0 then says any genuine match must have its column coordinate u' equal to 200 — it may slide up and down that vertical line, but it cannot leave it. This one scalar equation holds for every correct correspondence in the pair, which is why it is the quiet workhorse behind essentially all of two-view geometry.
A point in the left image is mapped by F to a line in the right image; the matching point lies on that line.
The Essential Matrix
The fundamental matrix is wonderfully undemanding — it asks for no calibration — but that very generosity hides something. Because F speaks in pixels, it mixes together two unrelated things: the internal quirks of each camera (focal length, principal point) and the external motion between the two viewpoints. If we already did the work of camera calibration in Guide 1 and so know the intrinsic matrix K, we can peel the camera quirks away and reveal the pure geometry underneath. That distilled object is the essential matrix E.
The trick is to stop measuring in pixels and start measuring in normalized coordinates. Multiplying a pixel x by the inverse intrinsics, x̂ = K⁻¹x, undoes the camera's internal scaling and re-expresses the point as a pure direction in the camera's own frame — as if every camera were an idealised unit-focal-length camera. In these clean coordinates the fundamental matrix becomes the essential matrix, and the epipolar constraint reads exactly the same way, x̂'ᵀ E x̂ = 0, just on normalized points. Whereas F was a soup of optics and motion, E is only motion.
Left: E is built from the rotation R and translation t between the cameras. Right: the bridge from F to E through the intrinsics.
Read the first equation, E = [t]×·R. Here R is the 3×3 rotation that re-aims camera 1's orientation into camera 2's — it answers 'how was the second camera turned relative to the first?'. The vector t is the translation between the two centres — 'how far, and in which direction, did the camera move?'. The symbol [t]× is the skew-symmetric matrix of t: a 3×3 matrix purpose-built so that multiplying by it reproduces the cross product, i.e. [t]× v = t × v for any vector v. We need a cross product because the epipolar plane is defined by two directions (the baseline t and the viewing ray R x̂), and the cross product is precisely the tool that tests whether three directions are coplanar. So E = [t]×R is just 'rotate the ray, then test coplanarity with the baseline' written as one matrix.
The skew-symmetric matrix of t = (tx, ty, tz): multiplying by it performs the cross product t × (·).
Don't be intimidated by the box of entries: it is just t's three components (tx, ty, tz) slotted into a fixed antisymmetric pattern with zeros on the diagonal. Plug in, say, t = (0, 0, 1) and multiply by a vector v = (vx, vy, vz): you get (−vy, vx, 0), which is exactly (0,0,1) × v. The matrix simply is the cross-product operation in disguise, so we can keep doing linear algebra. (One tidy consequence: because t × t = 0, the matrix [t]× is itself rank 2 — which is why E, and hence F, inherits rank 2.)
The second equation, E = K'ᵀ F K, is the bridge between the two worlds. K and K' are the intrinsic matrices of camera 1 and camera 2 (often the same camera, so K = K'). Sandwiching the pixel-space fundamental matrix F between the intrinsics converts it into the calibrated-space essential matrix — and the formula runs both ways, so if you ever recover E you can also produce F, and vice versa. This is the precise sense in which calibration unlocks metric motion: F alone can warn you where matches must lie, but only E, born from K, carries the actual rotation and translation of the camera. And that is the cliff-hanger for the final section — E can be cracked open to read off R and t, the camera's real movement through the world.
Estimating F from Point Matches
So far F has been a given. In practice we must compute it from data — and the only data we have is a list of point correspondences, the very feature matches produced by the detectors and descriptors from earlier computer-vision guides (think corners or SIFT/ORB keypoints matched between the two photos). The classic recipe is the eight-point algorithm, and its idea is delightfully clean: the epipolar constraint x'ᵀ F x = 0 is linear in the nine unknown entries of F, so each correspondence donates one linear equation, and a handful of correspondences pins F down.
Each match expands x'ᵀ F x = 0 into one row; stack the rows into A and solve A·f = 0.
Let us unpack the expansion. Write the unknown F as a flat 9-vector f = [F₁₁, F₁₂, …, F₃₃]ᵀ. Multiply out x'ᵀ F x = 0 with x = [u, v, 1]ᵀ and x' = [u', v', 1]ᵀ, and every term is a known pixel product times one unknown entry of f. Collecting them gives the single row shown above: its nine numbers are the pairwise products u'u, u'v, u', v'u, v'v, v', u, v, 1 — literally each coordinate of the second point times each coordinate of the first. One correspondence, one row. Stack the rows from many correspondences into a tall matrix A, and the whole system is A·f = 0. We seek the f that this matrix sends (closest) to zero.
Why eight points and not nine, when f has nine numbers? Because F is only defined up to scale — multiplying every entry by 5 describes the same epipolar lines, since x'ᵀ(5F)x = 0 is the same equation. That removes one degree of freedom, leaving eight to determine. We solve A·f = 0 not by inverting anything (the right-hand side is zero) but with the singular value decomposition (SVD) of A: the solution f is simply the singular vector belonging to the smallest singular value — the direction A squashes most nearly to zero. Treat SVD here as the standard, reliable tool for 'least-squares subject to a unit-length constraint'; we use it, we don't re-derive it.
There is one more enemy: outliers. Feature matchers are imperfect; some 'matches' are simply wrong, and a single bad correspondence can drag the least-squares F far off. The cure is RANSAC (RANdom SAmple Consensus): instead of trusting all matches, repeatedly gamble on a tiny random subset, fit a candidate F to it, and see how many of the other matches agree (their distance to the predicted epipolar line is small). The candidate with the largest agreeing crowd — the inliers — wins, and we refit F using only those. RANSAC is the immune system that lets the clean eight-point idea survive messy real photos.
import numpy as np
def normalize(pts):
"""Hartley normalization: zero mean, mean distance sqrt(2) from origin."""
c = pts.mean(axis=0) # centroid (cu, cv)
d = np.sqrt(((pts - c) ** 2).sum(1)).mean()
s = np.sqrt(2) / d # scale so mean distance = sqrt(2)
T = np.array([[s, 0, -s * c[0]],
[0, s, -s * c[1]],
[0, 0, 1]])
pts_h = np.c_[pts, np.ones(len(pts))] # to homogeneous
return (T @ pts_h.T).T, T
def eight_point(x1, x2):
"""Estimate F from >= 8 correspondences (x1[i] <-> x2[i])."""
x1n, T1 = normalize(x1)
x2n, T2 = normalize(x2)
# Build one row per match: [u'u, u'v, u', v'u, v'v, v', u, v, 1]
u, v = x1n[:, 0], x1n[:, 1]
up, vp = x2n[:, 0], x2n[:, 1]
A = np.column_stack([up*u, up*v, up, vp*u, vp*v, vp, u, v, np.ones(len(u))])
# Solve A f = 0: f is the smallest right-singular vector of A
_, _, Vt = np.linalg.svd(A)
F = Vt[-1].reshape(3, 3)
# Force rank 2 by zeroing the smallest singular value of F
U, S, Wt = np.linalg.svd(F)
S[-1] = 0
F = U @ np.diag(S) @ Wt
# Undo normalization so F works on raw pixels again
return T2.T @ F @ T1
def ransac_F(x1, x2, thresh=1.0, iters=2000):
"""Robust F: many random 8-point fits, keep the largest inlier set."""
best_F, best_inliers = None, []
n = len(x1)
for _ in range(iters):
idx = np.random.choice(n, 8, replace=False) # random minimal sample
F = eight_point(x1[idx], x2[idx])
inliers = [i for i in range(n)
if sampson_distance(F, x1[i], x2[i]) < thresh]
if len(inliers) > len(best_inliers):
best_F, best_inliers = F, inliers
return eight_point(x1[best_inliers], x2[best_inliers]), best_inliersRecovering Pose & Structure
We have F from pixels and, via K, the essential matrix E. Now we collect the prize. Recall E = [t]×R, so E secretly contains the camera's rotation and translation — and an SVD of E lets us factor them back out. The arithmetic is standard, but there is a subtlety worth knowing: the factorization is not unique. It yields four candidate (R, t) solutions — two possible rotations paired with two possible translation directions (t and −t). Geometrically these are the same baseline reflected and flipped in the natural ambiguous ways.
How do we pick the true one? With the cheirality check — a fancy word for a common-sense rule: a real 3D point must lie in front of both cameras, not behind them (you cannot photograph something behind your lens). For each of the four candidates, triangulate a test point and check the sign of its depth in both views. Only the correct (R, t) puts the point in front of both cameras; the other three place it behind one or both. One vote settles it.
With R and t finally fixed, we can rebuild the actual 3D points — the structure. The intuition is exactly the ray-intersection picture from stereo: each camera, knowing its pose, casts a ray through the matched pixel into space, and the two rays meet at the world point X. In the presence of noise the rays may not meet perfectly, so we solve for the X that comes closest to both. The standard linear method is the Direct Linear Transform (DLT) for triangulation.
DLT triangulation: each view contributes equations; stack them and solve for X by SVD.
Unpack it slowly. P = K[R | t] is a camera's 3×4 projection matrix — it staples the intrinsics K to the pose [R | t] from Guide 1, and it maps a homogeneous 3D point X = [X, Y, Z, 1]ᵀ to a homogeneous pixel. The symbol ~ means 'equal up to scale', because projection forgets depth: P·X gives the right pixel direction but an unknown overall scale factor. To kill that nuisance scale we use the cross product: if x and P·X point the same way, then x × (P·X) = 0 exactly. Each view's cross product gives two independent linear equations in the unknown entries of X (the third is redundant). Two views therefore stack into four equations — enough for the four homogeneous unknowns of X — assembled into a matrix A so that A·X = 0, and once again the smallest singular vector from SVD is our answer X.
Stand back and admire what two photographs just gave us: from nothing but a pile of pixel matches, we extracted the relative pose of the two cameras (R and t, direction only) and a sparse cloud of 3D points — the seed of a reconstruction. This is the heart of 3D reconstruction. But two views are only the beginning. In the next guide we chain many views together, share points across them, and jointly polish every camera and every point at once — Structure from Motion, SLAM, and bundle adjustment. The two-view machinery you built here is the atom from which those whole worlds are assembled.