multi-object tracking

Multi-object tracking (MOT) follows many targets at once and, crucially, assigns each a consistent identity (ID) that persists across the whole video, even as objects enter, leave, cross paths, and disappear behind one another. The output is not just boxes per frame but a set of trajectories: 'person 7' is the same person from frame 1 to frame 900. Keeping IDs stable through occlusion and clutter is what makes MOT genuinely hard, beyond detecting objects in each frame.

The dominant paradigm is tracking-by-detection: run an object detector on every frame, then perform data association to link new detections to existing tracks. Association scores each detection-to-track pair by motion agreement (does the detection fall where a Kalman filter predicted?) and appearance similarity (does it match the track's learned embedding?), then solves the global assignment optimally with the Hungarian algorithm. The lineage is instructive: SORT uses only IoU + Kalman + Hungarian and is extremely fast; DeepSORT adds a deep re-identification embedding so an ID survives short occlusions; ByteTrack's insight is to also associate low-confidence detections (often partially occluded objects) instead of discarding them, sharply reducing lost tracks. End-to-end transformer trackers (TrackFormer, MOTR) fold detection and association into one network using track queries.

Evaluation is its own subject because there are two failure modes to balance: detection errors (misses, false positives) and association errors (ID switches, where two trajectories swap labels, and fragmentations). MOTA emphasizes detection, IDF1 emphasizes identity preservation, and the modern HOTA metric explicitly decomposes and balances detection accuracy against association accuracy. Standard benchmarks are MOTChallenge (MOT16/17/20, pedestrians in crowds) and KITTI/BDD100K for driving. The recurring enemies are crowded scenes, long occlusions, and visually identical targets (a group of similarly dressed people).