video understanding

Video understanding means reasoning not over one frame but a sequence of them through time — a cooking tutorial, a security clip, a sports highlight, a lecture recording. The model must grasp what is happening and how it changes: who entered the room, what step comes next, why the player was offside. It is vision plus the extra dimension of before and after.

The naive approach samples frames — a handful per second, or keyframes — encodes each like an image, and feeds the whole stream of image tokens to the LLM, often with timestamps so it can reason about order and duration. The challenge is sheer volume: a minute of video at even modest sampling is thousands of tokens, so systems pool frames, pick salient moments, or use memory tricks to fit a long clip into the context window. Audio is frequently fused in too, since sound carries much of a video's meaning.

Because most frames are skipped, models miss brief events between samples and struggle with fine temporal order — did she add salt before or after the egg? Long videos remain largely out of reach without aggressive summarisation.