two-stream network

A two-stream network splits video understanding into two parallel CNNs that mirror a rough division of perception: one stream looks at appearance (a single RGB frame, answering 'what objects and scene are present?') and the other looks at motion (a stack of optical-flow fields, answering 'how are things moving?'). Their predictions are then fused. The idea, from Simonyan and Zisserman (2014), was that explicit motion input would supply what a 2D image network alone could not see, and it was the architecture that first made deep video classification competitive.

The spatial stream is a standard image CNN run on one (or a few sampled) RGB frames, capturing object, scene, and pose cues, which is also why it can be pretrained on ImageNet. The temporal stream takes a stack of L consecutive optical-flow fields (each a two-channel image of horizontal and vertical displacement, so 2L input channels), so its first layer directly sees short-term motion. The two streams are trained separately and combined by late fusion (averaging softmax scores or training an SVM), though later work fused them at mid-level convolutional layers for richer interaction. Temporal Segment Networks (TSN) extended the idea to long videos by sampling segments spread across the whole clip and aggregating, addressing the original method's short temporal coverage.

Its lasting lesson is that motion (flow) and appearance are complementary, and feeding pre-computed optical flow gave a large accuracy boost that pure-RGB networks of the era could not match. Its lasting problem is cost: computing optical flow is slow and storage-heavy, breaking real-time use. This drove two responses, learning the motion representation inside the network (hidden two-stream, TVNet) and abandoning explicit flow for 3D convolutions that learn motion implicitly. Even the strongest 3D model of its day, two-stream I3D, kept the flow stream because it still helped.