From Stills to Streams: How Pixels Move

A video is more than a stack of images

Until now you have studied a single image at a time: a still grid of pixels frozen forever. A video breaks that spell. It is simply a sequence of still images, called frames, captured one after another at a steady pace. The pace is the frame rate, measured in frames per second (fps). A movie at 24 fps shows you 24 separate photographs every second; your phone might shoot 30 or 60. A short stretch of such frames — a few seconds you actually work with — is called a clip.

The cleanest way to picture this is a flipbook: a stack of drawings, each slightly different from the one below it, that turns into smooth motion when you riffle through them. Every page is an ordinary picture; the magic lives in the order and the timing. This gives us a powerful new mental model. A single image has two axes, width (x) and height (y). A video adds a third axis: time (t). So a video is a little 3D block of numbers — a stack of pixel grids — and the new axis we just unlocked is the temporal dimension.

Now the two opposing facts that secretly define this entire field. Fact one: temporal redundancy. Two consecutive frames look almost identical. If nothing in the world moved much in 1/30 of a second, the vast majority of pixels keep the exact same value. This is wasteful in one sense — but it is also a gift, because it is precisely why video compresses so well: we can store one frame and then only the small changes for the next. Fact two: temporal information. The tiny set of pixels that do change between frames is exactly the part we care about. A walking person, an opening door, a thrown ball — all of it lives entirely in those small differences.

Everything in this track is one long answer to a single question: how do we understand change over time? That skill has a name — temporal modeling — and it is the thread running through every guide here. By the end you will go from "a pixel changed" all the way up to systems that watch a clip and name the action inside it, a task called video classification. We start, in this guide, at the very bottom of that ladder, with no math at all — just the idea that time is the new axis and change is the prize.

Frame differencing: the simplest motion detector

Let us build a working motion detector right now, with one idea: if a pixel's value changed from the last frame to this one, something probably moved there. That is the whole algorithm. Take the current frame, subtract the previous frame pixel-by-pixel, take the absolute value of the difference, and call any pixel whose change is big enough "moving." This is called frame differencing, and despite its simplicity it is a genuine, crude form of motion estimation — our first attempt to turn raw pixels into a statement about motion.

Let us read this slowly, symbol by symbol. I_t(x,y) is the brightness of the pixel at column x, row y, in the frame at time t; I_{t-1}(x,y) is that same pixel one frame earlier. Their difference is how much that one spot changed. The vertical bars |\cdot| are absolute value: they throw away the sign, so a pixel getting brighter (a positive change) and a pixel getting darker (a negative change) both count as motion — we care that it changed, not which way. The result D_t(x,y) is the difference image: a new grid where big numbers mark big changes. The second equation turns that into a yes/no answer. The brackets [\cdot] are the Iverson bracket: [\text{statement}] equals 1 when the statement is true and 0 when it is false. So M_t(x,y) is 1 (moving) wherever the change exceeds the threshold \tau, and 0 (still) otherwise. M_t is our foreground mask — a clean black-and-white stencil of where motion is.

Numbers make this real. Imagine one tiny 3x3 patch of an 8-bit grayscale image (values 0–255). At time t-1 the patch is flat, value 50 everywhere; at time t a bright edge has slid across its right column:

I_{t-1}            I_t              D_t = |I_t - I_{t-1}|     M_t = [D_t > tau], tau = 30
50 50 50          50 51 200        0   1   150               0  0  1
50 50 50          49 50 210        1   0   160               0  0  1
50 50 50          51 49 205        1   1   155               0  0  1

# Left column barely moved (0 or 1) -> sensor noise, stays 0 in the mask.
# Right column jumped ~150 -> the moving edge lights up as 1.
# A threshold of tau = 30 cleanly separates the noise from the real motion.

Why threshold at all? Because of \tau's real job. A camera sensor never reports the exact same number twice even for a perfectly still scene — tiny electrical fluctuations make a 50 read as 49 or 51. Without a threshold, every pixel would flicker "moving" and the mask would be useless static. \tau is the dial that says "ignore changes smaller than this." Choosing it is a trade-off: set \tau too low and noise leaks through as false positives (phantom motion everywhere); set \tau too high and you become deaf to genuine but subtle motion, missing slow or low-contrast movers. In our example \tau = 30 sits comfortably above the noise (changes of 0–1) and well below the real edge (changes of ~150).

Background subtraction: separating movers from the scene

Frame differencing compares this frame to the previous one — a memory only one frame deep. That short memory is exactly what causes holes and ghosting: it has no idea what the scene is supposed to look like when nothing is happening. The upgrade is to compare each frame not to its neighbor but to a learned model of the static scene itself. This is background subtraction, and it is the workhorse of fixed-camera surveillance. The intuition: every pixel keeps a running memory of its usual color — the empty hallway, the bare parking space — and a foreground object is simply whatever deviates strongly from that memory.

The simplest such memory is a running-average background. We keep an estimate B_t of what the background looks like, and gently nudge it toward each new frame:

Symbol by symbol: B_t(x,y) is our current best guess of the background brightness at pixel (x,y); B_{t-1}(x,y) is the guess we had a frame ago; I_t(x,y) is the brightness we just observed. The number \alpha (alpha), somewhere in [0,1], is the learning rate — the weight we give the newest frame. Read the formula in plain words: "the new background is a blend of a little bit of what I just saw and a lot of what I already believed." If \alpha = 0.05, then each update is 5% new observation and 95% old memory. A concrete step: if my stored background at a pixel is B_{t-1} = 100 and the new frame reads I_t = 200 (a person just walked there), then B_t = 0.05\times 200 + 0.95\times 100 = 10 + 95 = 105. The background barely twitched — it refuses to be fooled by one bright frame.

So \alpha literally means "how much do I trust the newest frame?" A small \alpha gives a long memory: the background adapts slowly, robustly ignoring objects that pass through briefly — but it is sluggish to react when the scene genuinely changes (a parked car that leaves). A large \alpha gives a short memory: it adapts fast to real change, but a person who pauses for a few seconds will slowly "bake into" the background and vanish from your foreground. Detection itself is then just a difference-and-threshold against this model: |I_t(x,y) - B_t(x,y)| > \tau flags foreground — the same idea as before, but now measured against the scene's true resting state instead of one arbitrary previous frame, which is exactly what fills in the holes.

# Running-average background subtraction (one channel, fixed camera)
import numpy as np

alpha = 0.05         # learning rate: small = long memory
tau   = 30           # foreground threshold
background = first_frame.astype(np.float32)   # initialize memory

for frame in video:
    f = frame.astype(np.float32)
    diff = np.abs(f - background)             # deviation from the scene's usual look
    mask = (diff > tau).astype(np.uint8)      # 1 = foreground (a mover), 0 = background
    background = alpha * f + (1 - alpha) * background   # update the memory
    yield mask

A single average per pixel is a fragile model, because it assumes each pixel has one usual color. The real world breaks that constantly: leaves swaying in wind make a pixel flip between green-leaf and blue-sky many times a second, and a flickering screen or rippling water does the same. The average lands uselessly in the middle and flags the natural flicker as foreground. The fix is to let a pixel remember several usual colors at once. A per-pixel Gaussian models a pixel's color as a bell curve — a mean plus an allowed spread — so it tolerates a little jitter. The Mixture of Gaussians (MOG) goes further: it keeps several bell curves per pixel (one for "leaf," one for "sky") and counts you as foreground only if you match none of them. That is why MOG is the classic answer for multimodal backgrounds.

From 'something moved' to 'how it moved'

Everything so far answers one yes/no question: did this pixel move? That is detection, and its output is a binary mask. But to truly understand motion we want estimation — not just whether a spot moved, but how it moved: how far, and in which direction. That richer answer is a displacement vector at every location, and the field of all those little arrows across the image is called a motion field. Promoting a mask to a motion field is the jump from "something is happening here" to "this object slid two pixels left and one pixel up."

The most intuitive way to estimate that vector is block matching, and it works exactly the way you would search for something by eye. Take a small block — a patch — around a point in frame t. Then in the next frame t+1, slide that patch around a small search window, trying each candidate offset, and ask: "at which shift does the patch line up best with the new frame?" The winning shift is your estimated motion for that block. You are literally asking "where did this piece go?" and trusting the best-matching offset.

Here is the cost we minimize, symbol by symbol. (u,v) is a candidate displacement — a guess that the patch shifted u pixels horizontally and v pixels vertically. The sum runs over every pixel (x,y) inside the patch. For each one we compare its value in frame t, namely I_t(x,y), with the value at the shifted spot in the next frame, I_{t+1}(x+u, y+v). We take their difference, square it (so any mismatch is positive and large errors are punished hard), and add up these squared errors over the whole patch. That total is the SSD — the Sum of Squared Differences — a single number measuring how badly the patch fits at shift (u,v): small SSD means a great match, large SSD a poor one. The second piece, \operatorname{arg\,min}, just means "the (u,v) that makes SSD smallest"; that winning shift (\hat{u},\hat{v}) is our estimated motion. A concrete sketch: a patch sits at SSD = 4000 if you do not shift it, 120 if you shift it (+2, -1), and 5000 if you shift it (-3, 0); the offset (+2, -1) wins, so we report "this block moved two pixels right and one pixel up."

Pitfalls, and the road ahead

Before we move on, one conceptual distinction that will save you endless confusion: apparent motion in the image is not the same as real 3D motion in the world. Our cameras only ever see brightness patterns shifting on a flat sensor; from that, we infer motion. The two usually agree, but not always — a slowly rotating barber's pole appears to move upward forever, and a uniform wall can move without any pixel changing at all. Everything we compute is apparent motion, and we must stay humble about the gap between the pixels' story and the world's truth.

Let us also gather the limits of the simple tools we just built, because each one is really a signpost to a better method. Frame differencing leaves holes in smooth objects and ghosts of fast ones. Background subtraction conquers the holes but stumbles on sudden lighting changes and on shadows that masquerade as movers. Block matching gives us real motion vectors but trips on the aperture problem along smooth edges. None of these is a dead end — each failure is the precise reason the next technique exists.

So here is the map of where this track is taking you. Guide 2 turns block matching's rough arrows into precise, dense, per-pixel motion and confronts the aperture problem directly — that is optical flow. Guide 3 stops looking frame-to-frame and instead follows one chosen object persistently through a whole clip, even when it is briefly hidden — that is object tracking. Guide 4 climbs from where-and-how-things-move up to what is happening — teaching networks to watch a clip and name the action, the realm of video classification and action recognition. And Guide 5 goes dense and generative: labeling every pixel of every frame over time, and even inventing the frames in between (segmentation and interpolation).

You started this guide with a flipbook and a single subtraction, and you already hold the core insight that powers the whole field: in video, the prize is change, and change is a faint signal we must carefully separate from a noisy, mostly-static, possibly-moving-camera world. Every later guide is just a sharper, smarter tool for the same hunt. Onward to measuring motion pixel by pixel.