action recognition

Action recognition is the flagship instance of video understanding: identify what a person (or people) is doing, such as 'jumping', 'shaking hands', or 'opening a door'. It is essentially video classification specialized to human activities, and for many researchers the two terms are used almost interchangeably because action benchmarks have driven most progress in video models.

It is important to separate three task settings that are often conflated. Trimmed clip classification assumes the video already contains one action and just names it. Temporal action detection (localization) works on untrimmed video and must say both what action occurs and when (its start and end times). Spatio-temporal action detection (e.g., the AVA benchmark) goes further, localizing the actor in space and time, producing a tube of boxes plus a label. The harder settings need motion modeling that respects temporal order: distinguishing 'opening' from 'closing' a drawer is impossible without knowing the direction of motion, which is exactly where appearance-only models fail.

Methods mirror the video-model timeline: two-stream networks, 3D CNNs and I3D, the SlowFast network (a slow high-resolution appearance pathway plus a fast low-resolution motion pathway), and video transformers. A distinct family is skeleton-based recognition (ST-GCN and successors), which first extracts human pose and runs a graph convolution over joints, gaining robustness to background and appearance at the cost of needing reliable pose estimation. Core benchmarks include UCF101 and HMDB51 (older, appearance-heavy), Kinetics-400/600/700, Something-Something (temporal reasoning), AVA (spatio-temporal), and Charades (multi-label daily activities).