temporal modeling

Temporal modeling is the general problem of capturing how information depends across time in a sequence such as video. The crux is that meaning lives in the ordering and the relations between moments: adjacent frames carry short-range motion (a foot pushing off, a ball leaving a hand), while distant frames carry long-range structure (the overall arc of 'making a cup of tea' that unfolds over minutes). A good temporal model must handle both scales, and most architectures are strong at one and weak at the other.

Three mechanisms dominate. Recurrence (RNN, LSTM, GRU) processes frame features one at a time, carrying a hidden state forward, which is naturally sequential and can in principle reach far back but suffers vanishing gradients and cannot parallelize over time. 3D convolution (and temporal convolution) has a local temporal receptive field that only grows with depth, making it efficient and parallel but biased toward short-range patterns. Self-attention (transformers) compares every timestep with every other directly, capturing arbitrary-distance dependencies in one step, at the price of cost that grows quadratically with sequence length. Cheap hybrids exist too, like the Temporal Shift Module (TSM), which shifts a fraction of channels along time inside a 2D CNN to get temporal mixing for nearly free.

The genuinely hard part is long-range temporal modeling. Because clips are usually only a few seconds, most models never see the global structure of a long activity, and quadratic attention cost or limited convolutional receptive fields cap how far they can reason. This motivates hierarchical and memory-based designs: temporal pyramids, video transformers with factorized or windowed attention to extend range affordably, and external memory that stores compressed past context, all attempts to reason over minutes, not just moments, without exploding compute.