The Shape and Look of a Face: Eigenfaces, Fisherfaces, and Deformable Models

A face is a point in a million-dimensional space

Here is a mental flip that makes everything in this guide click. Take a small grayscale photo, say 100 by 100 pixels. That is 100 × 100 = 10,000 numbers, each one a brightness value from 0 (black) to 255 (white). Now stop thinking of those 10,000 numbers as a little grid you look at, and start thinking of them as a single list — one long column of 10,000 entries. A list of 10,000 numbers is exactly a point in a 10,000-dimensional space, the same way a list of 2 numbers (x, y) is a point on a sheet of paper and a list of 3 numbers is a point in the room. Bump one pixel brighter and you have moved a tiny step along one of those 10,000 axes. A real 1-megapixel face would live in a literally million-dimensional space.

That space is unimaginably vast, and here is the crucial fact: almost none of it looks like anything. Pick 10,000 numbers completely at random and the 'image' you get is pure snow — the visual hiss of an untuned analog TV. Faces are not scattered all over this space. They cluster in an astonishingly thin, smooth region of it. Move a little in some directions and a face stays a face (slightly rounder cheeks, a hint more smile); move the same distance in almost every other direction and the face instantly dissolves into static. That thin, curved region where real faces live is called the face subspace, or the face manifold.

This reframing motivates everything that follows. If a face really only varies along a few directions, then we should be able to compress it from 10,000 numbers down to maybe 50, with almost no visible loss — and recognition becomes much easier in that tidy little space than in the raw pixel jungle. This is the core idea behind the subspace methods we will build: eigenfaces and its cousins. It also slots neatly into the classical recognition pipeline you already know: we are simply choosing a smarter feature representation (a short code) before handing it to a classifier, instead of comparing raw pixels.

Eigenfaces: principal component analysis for faces

Eigenfaces answers the question 'what are the few directions along which faces vary the most?' using a classic statistical tool: principal component analysis (PCA). The picture to keep in your head is a recipe. Start from one average face — the blurry, generic face you get by literally averaging a big pile of training photos pixel by pixel. Then keep a small set of 'ingredient faces', and to reconstruct any particular person you add a weighted dose of each ingredient on top of the average: average face + 3.1 × (ingredient₁) − 1.7 × (ingredient₂) + 0.4 × (ingredient₃) + … . Those weights — just a few dozen numbers — are that person's entire face code. The ingredient faces themselves are the eigenfaces, and they famously look like pale, ghostly, half-formed faces.

# Eigenfaces = PCA on a stack of face images
# X: shape (N, D) -- N training faces, each flattened to D = H*W pixels
mean_face = X.mean(axis=0)        # the "average face", a length-D vector
A = X - mean_face                 # centre every face: subtract the mean

# When D >> N (e.g. 10000 pixels, 300 faces), do NOT build the D x D
# covariance. Build the small N x N Gram matrix instead -- same eigenfaces,
# vastly cheaper. (See the 'Gram trick' note below.)
G = (A @ A.T) / N                 # shape (N, N)
eigvals, V = eig(G)               # sort columns of V by eigvals, descending

# Map the small eigenvectors back to image space -> the eigenfaces
U = A.T @ V                       # shape (D, N): each column is an eigenface
U = U / norm(U, axis=0)           # normalise each eigenface to unit length

# Keep only the top-k directions (largest eigenvalues)
Uk = U[:, :k]                     # shape (D, k)

def encode(face):                 # turn a 10000-pixel face into a k-number code
    return Uk.T @ (face - mean_face)   # y: the compact "face code"

def reconstruct(y):               # rebuild an approximate face from its code
    return mean_face + Uk @ y

The covariance of the centred faces (left), and its eigen-decomposition (right).

Let us unpack the left equation symbol by symbol. Each x_n is the n-th training face written as a tall column vector of D pixels; N is how many training faces we have; and x̄ (read 'x-bar') is the mean face, the pixel-by-pixel average. The term (x_n − x̄) is one face with the average subtracted off, so it only carries what makes that face different from the norm. Multiplying a D-long column by its own transpose, (x_n − x̄)(x_n − x̄)ᵀ, produces a D×D table whose (i, j) entry says 'when pixel i is above average, does pixel j tend to be above average too?'. Averaging that table over all N faces gives the covariance matrix C: a complete summary of how the pixels vary together. The right equation, C·u_k = λ_k·u_k, is the eigen-decomposition: it asks for special directions u_k that C only stretches, never rotates. Those directions u_k are the eigenfaces, and the eigenvalue λ_k is exactly how much the faces vary along u_k (its variance). A big λ means 'faces spread out a lot in this direction — it is informative'; a tiny λ means 'faces barely move here — it is noise'. Concretely, if λ₁ = 50,000 and λ₅₀ = 12 and λ₅₁ = 0.3, plotting the sorted λ values (the eigenvalue spectrum) shows a steep drop, and you simply keep components down to where the curve flattens — often 30 to 150 of them explain ~95% of the total variance.

Projecting a face onto the top-k eigenfaces to get its compact code y.

This last line is how we compress a face. U_k is the D×k matrix whose columns are the top k eigenfaces (the most informative directions, ordered by descending λ). We first centre the new face by subtracting x̄, then U_kᵀ·(x − x̄) measures how much of each eigenface that face contains — it dots the face against each ingredient. The result y is just k numbers (say 50), the 'weights' from our recipe. We keep the large-λ directions precisely because they carry the most variation, so this short code preserves almost everything that distinguishes one face from another while discarding the noise. To go backwards, reconstruct ≈ x̄ + U_k·y rebuilds an approximate face from its code; keep more components and the reconstruction sharpens. Recognition is now wonderfully cheap: store each known person's code y, encode a new photo, and find the closest stored code. That is exactly the k-nearest-neighbors image classifier from Guide 2 — but running in a clean 50-dimensional face space instead of 10,000 raw pixels, so it is faster and far more robust. One practical caution lives in the code above: building the D×D covariance C when D = 10,000 is a 100-million-entry monster, yet with only N ≈ 300 faces it is mostly redundant. The Gram-matrix trick computes the small N×N matrix (1/N)·AᵀA-style instead and maps its eigenvectors back to image space, giving identical eigenfaces for a fraction of the cost.

Fisherfaces: tell people apart, not lighting

Eigenfaces has a sharp flaw, and naming it teaches the next idea. PCA finds the directions of greatest total variance — but it does not know or care what that variance means. In real face datasets, the single biggest source of pixel variation is usually not who the person is; it is the lighting (a face lit from the left versus the right looks wildly different pixel-for-pixel) and the expression. So PCA happily spends its first, most powerful eigenfaces describing lighting, and may bury the subtle who-is-this signal in components it discards as 'small'. The unsettling consequence: an eigenface system can decide two photos are far apart just because the lamp moved, even though it is the same person.

Fisherfaces fixes this by changing the goal from 'capture the most variation' to 'capture the variation that separates people'. The tool is Linear Discriminant Analysis (LDA), Fisher's classic idea. Picture two clouds of dots on a page, one cloud per person (each dot is one of that person's photos). PCA would project onto the axis along which all the dots, lumped together, spread out the most — which might run straight through both clouds and smear them together. LDA instead hunts for the axis onto which, when you cast the shadows of the dots, the two clouds land as two tight, well-separated clumps. It does this by maximizing the spread BETWEEN the cloud centres while minimizing the spread WITHIN each cloud. The resulting directions, the fisherfaces, are far more robust to lighting because lighting changes are within-class wobble (the same person under different lamps) that LDA actively tries to suppress.

Fisher's criterion (left) and the generalized eigenproblem that maximizes it (right).

Read J(w) as a 'separation score' for a candidate projection direction w (think of w as a single arrow we cast shadows onto). Two ingredients go into the score. S_B is the between-class scatter matrix: it measures how far each person's cloud-centre sits from the global average face — big S_B means the people are well separated. S_W is the within-class scatter matrix: it measures how spread out each person's own photos are around that person's own centre — big S_W means messy, smeared clouds. The expression wᵀS_B w is the between-class spread seen along direction w, and wᵀS_W w is the within-class spread along w. We want the numerator big (people far apart) and the denominator small (each person tight), so we maximize their ratio J(w). Tiny example: if along some w the class means project to 0 and 10 (gap 10, so S_B-term ≈ 100) while each person's photos scatter by only ±0.5 (S_W-term ≈ 0.25), then J ≈ 400 — excellent. Along the lighting axis the gap might be 1 but the within-person scatter 8, giving J ≈ 0.125 — terrible, exactly the axis we want to avoid. Maximizing this ratio turns out to be the generalized eigenproblem S_B·w = λ·S_W·w on the right, whose top eigenvectors are the fisherfaces.

Shapes that bend in learned ways: active shape models

Eigenfaces and fisherfaces model a face's appearance — its pixel brightnesses. But a face also has geometry: the layout of its parts. The active shape model (ASM) captures that. We mark a fixed set of landmarks on every training face — say 68 agreed points at the corners of the eyes, along the mouth, around the nose, and tracing the jaw. Each landmark has an (x, y) position, so 68 of them stack into one 'shape vector' of 136 numbers. Just as a face was a point in pixel space, a shape is now a point in this 136-dimensional shape space, and a real human face's landmark layout lives in only a thin region of it — most random shapes look like a tangle, not a face.

There is one wrinkle to handle first. Two photos of the identical face can give different shape vectors simply because the head is bigger, shifted, or tilted in one of them — differences of position, scale, and rotation that have nothing to do with the face's actual shape. So before learning, we align all the training shapes to a common frame, sliding, scaling, and rotating each one to match as closely as possible. This alignment step is called Procrustes analysis (named after the mythical Greek who stretched or trimmed guests to fit his bed). Once the shapes are aligned, we run the same PCA machinery from the eigenfaces section — but on shape vectors instead of pixel vectors. The output is a Point Distribution Model: a mean shape plus a few modes of variation. Each mode is a direction in shape space that, reassuringly, corresponds to something meaningful learned from the data — mode 1 might open and close the mouth, mode 2 might turn the head left-right, mode 3 might widen the jaw.

The point distribution model (left) and the plausibility constraint on each shape parameter (right).

The left equation is a face-shape generator. x is the shape vector it produces — all 136 landmark coordinates of one face. x̄ is the mean shape, the average landmark layout. P is the matrix whose columns are the shape eigenvectors, i.e. the modes of variation (say the top 10 directions PCA found). And b is the short list of shape parameters — one number per mode, saying how much of that mode to add. Set b = 0 and you get exactly the mean face; set b = (3, 0, 0, …) and, if mode 1 is 'mouth opening', you get the mean face with a wide-open mouth. So a whole face shape is summarized by maybe 10 numbers in b. The right inequality is what keeps the generator honest. Recall λ_i is the variance of mode i across the training set, so √λ_i is its standard deviation. The constraint |b_i| ≤ 3·√λ_i caps each parameter at ±3 standard deviations — within the range the training faces actually explored. Concretely, if mode 1 had √λ₁ = 4, then b₁ is allowed to roam in [−12, +12] but no further. This is the heart of ASM: the model can bend, but only in the directions and by the amounts it saw real faces bend during training. Push b past the limit and the shape would deform into something no human face ever does; clamping b keeps every generated shape plausible.

1. Initialize: place the mean shape x̄ roughly over the face in the image (e.g. from a face detector's box from Guide 3).
2. Search locally: at each landmark, look a short distance along the line perpendicular (normal) to the shape boundary for the strongest nearby edge — a better guess for where that point should sit.
3. Propose: move every landmark to its best new edge location, forming a raw proposed shape (which may now look slightly inhuman).
4. Project back to the subspace: fit the model x = x̄ + P·b to the proposal and clamp each |b_i| ≤ 3·√λ_i, snapping the shape back to the nearest plausible face.
5. Repeat steps 2–4 until the landmarks stop moving (convergence).

Adding texture: active appearance models

An active shape model knows where a face's parts are, but not what they look like — it tracks the outline, not the skin, the eye colour, or the shadows. The active appearance model (AAM) completes the picture by modelling appearance too, and the trick that makes it work is a clean separation of shape from texture. For each training face, AAM warps its pixels so that its landmarks land exactly on the mean shape — geometrically 'rubber-sheeting' every face into the same standard frame. Now every warped face has the same shape, so whatever pixel differences remain are pure texture (skin tone, beard, lighting on the cheeks), with the geometry factored out. We have cleanly split one face into two independent descriptions: its shape (where the points are) and its texture (the colours inside the standard frame).

From here AAM does the now-familiar move: run PCA on the shapes (giving shape parameters, just like ASM) and separately run PCA on the shape-free textures (giving texture parameters), then combine the two into a single set of appearance parameters. The result is a generative model of the whole face: feed it a short vector of parameters and it synthesizes a complete, photo-realistic face image — both the right outline AND the right pixels filling it in. Eigenfaces could only blur faces together; AAM can render a sharp, specific face from a handful of numbers, because it models geometry and texture separately and recombines them.

Fitting an AAM: minimize the squared pixel difference between the warped real image and the model's synthesized face.

Fitting is the act of finding the parameters that make the synthesized face match a real photo. Here p is the full bundle of parameters (shape + texture + pose). A_model(p) is the face the model renders from those parameters — its best guess at the appearance. warp(x; p) takes the model's predicted shape and warps the real input image onto the standard frame, and I_image(warp(x; p)) is the real pixels sampled there. Their difference, the residual r(p), is just 'model's face minus the real face', pixel by pixel — when it is near zero everywhere, the model has locked onto the face. We square and sum the residual (the ‖·‖² ) and search for the p that makes it smallest: an iterative least-squares minimization, nudging the parameters, re-rendering, measuring the new error, and repeating. The classic AAM speed trick is to precompute, once, how a given pattern of pixel error should translate into a parameter update (a fixed linear update / Jacobian learned offline), so each iteration is a cheap matrix multiply instead of an expensive re-optimization — which is what made AAMs fast enough for real-time face tracking.

From one object to many parts: a deformable teaser

Step back and notice the shared assumption running through this whole guide. Eigenfaces, fisherfaces, ASM, and AAM are all powerful, but every one of them assumes a single, roughly-aligned object class: faces, already found and boxed, all looking forward in more or less the same frame. They model how one known thing varies. They have no good answer to 'somewhere in this messy street scene, find every pedestrian and every car'.

General object detection needs two things these models lack. First, the model must slide over an entire scene at many positions and scales, the way the face detector in Guide 3 swept a window across the image — not start from a tidy pre-aligned crop. Second, and this is the new ingredient, its parts must be able to move relative to each other: a walking person's arms, legs, and head are not in fixed positions the way a frontal face's eyes are. A rigid template snaps; we need a model whose parts can shift on flexible connections while staying recognizably one object.

That is precisely the deformable part model (DPM), the subject of Guide 5, and it is best understood as the synthesis of everything in this track. It takes a root template for the whole object plus several smaller part templates, connected to the root by 'springs' that penalize parts straying from their expected place — so the object can deform, but only sensibly, echoing the learned-deformation idea from this very guide. The templates are built from the gradient-orientation features (HOG) of Guide 2, the whole thing slides over the image and scores windows in the detection style of Guide 3, and it is trained with a latent SVM that figures out where the parts belong without being told. Features, detection, and deformable geometry, finally combined into one detector — that is where we go next.