video classification

Video classification assigns a single label to an entire clip, for example tagging a few-second clip as 'playing guitar' or 'diving'. The defining challenge versus image classification is temporal aggregation: a video is dozens to hundreds of frames, and the model must summarize evidence across time into one decision rather than judging a single picture.

Approaches span a spectrum of how seriously they model time. The cheapest treat the video as a bag of frames: run a 2D CNN per frame and average the predictions (late temporal pooling), which is a surprisingly strong baseline. Stronger temporal models include recurrent stacks (CNN features fed into an LSTM), two-stream networks (separate appearance and optical-flow branches), 3D CNNs that convolve over space and time, and modern video transformers (TimeSformer, ViViT, VideoMAE) that apply self-attention across spacetime tokens. Standard datasets are Kinetics (the de facto pretraining set), Sports-1M, and Moments in Time.

A subtle but central issue is scene and appearance bias. For many benchmark clips a single frame is nearly sufficient, because the background or objects (a basketball court, a guitar) give away the label without any motion understanding. This invites shortcut learning, where a model scores well by recognizing context rather than the action itself. It is why the Something-Something dataset was constructed to be unsolvable from one frame (it requires understanding the temporal direction of an interaction), and why reporting strong numbers on appearance-heavy benchmarks does not prove a model understands motion.