background subtraction

Background subtraction exploits a simple setup, a fixed camera watching a mostly-static scene, to find moving objects almost for free: build a model of what the empty background looks like, then flag any pixel that currently deviates from it as foreground. The intuition is that of a security camera on a quiet street, anything that suddenly differs from the learned 'normal' view (a passing car, a walking person) stands out as motion against an unchanging backdrop.

The core engineering is how each pixel's background is modeled and kept up to date. The simplest is a running average or median over recent frames, with a fixed threshold on the difference. The classic robust approach is the Mixture of Gaussians (MOG/MOG2, Stauffer-Grimson): each pixel's color is modeled as several Gaussian components, so a pixel can have more than one legitimate background appearance, which is essential for multimodal backgrounds like swaying trees, rippling water, or a flickering monitor. Other strong methods include KNN models and sample-based ViBe, and recent work uses learned networks. Critically, all of these adapt over time via a learning rate, slowly folding persistent changes (gradual lighting, a parked car that stays) into the background so the model does not become stale.

The raw foreground mask is then cleaned with morphological operations (to remove speckle and fill holes) and connected-component analysis to extract object blobs. The well-known failure modes define the field's difficulty: cast shadows are darker but move, so they are wrongly labeled foreground unless a shadow-removal step is added; sudden illumination changes (a cloud, lights switched on) flip large regions to foreground; 'ghosts' appear when a long-static object finally moves; camouflage (a foreground object the same color as the background) is missed; and any camera shake violates the static-camera premise and requires registration first. Despite this, it remains a workhorse for surveillance, traffic monitoring, and people counting because it is cheap and real-time.