3d convolutional network

A 3D convolutional network extends ordinary convolution from two dimensions to three by sliding kernels over time as well as over height and width. Where a 2D filter sees a small spatial patch of one frame, a 3D filter (say 3×3×3) sees a small spacetime volume spanning a few frames, so it can directly learn motion patterns, the way an edge moves or a limb swings, jointly with appearance, in a single unified operation rather than via a separate flow stream.

Concretely the input is a clip tensor of shape time × height × width × channels, and each kernel has a temporal extent k_t alongside spatial extents k_h and k_w; the network stacks such layers with 3D pooling. C3D (Tran et al., 2015) popularized this with simple 3×3×3 kernels throughout. The cost is steep: adding a time dimension multiplies the parameters and the compute relative to a comparable 2D network, making 3D CNNs data-hungry and slow to train. A key efficiency idea is factorization: R(2+1)D splits each 3D convolution into a spatial 2D convolution followed by a temporal 1D convolution, which cuts parameters, adds an extra nonlinearity (improving accuracy), and is easier to optimize; Pseudo-3D (P3D) and S3D use related decompositions.

The practical bottleneck shaped the field's history. Trained from scratch, 3D CNNs needed enormous labeled video (Sports-1M, then Kinetics) to beat 2D-plus-motion methods, because they have far more parameters and no convenient pretraining. This is precisely the gap that inflation (I3D) closed by bootstrapping 3D filters from ImageNet-pretrained 2D networks, and later efficient designs (X3D, which expands a small network along axes like frames, resolution, width; and SlowFast) made spatiotemporal convolution both accurate and affordable.