depthwise separable convolution

A depthwise separable convolution factors one ordinary convolution into two cheaper steps, by separating 'where' from 'what'. First a depthwise step filters each input channel on its own with its own little 2D kernel — it looks at spatial neighbourhoods but never mixes channels. Then a pointwise (1x1) step mixes the channels at each position but looks at no neighbourhood. A standard convolution does both jobs at once in one expensive operation; splitting them keeps almost the same modeling power for a fraction of the cost.

Precisely, a standard kxk convolution from C_in to C_out channels costs H*W*C_in*C_out*k*k multiply-adds and C_in*C_out*k*k parameters. The depthwise step costs H*W*C_in*k*k (one k x k kernel per input channel, no channel mixing); the pointwise step costs H*W*C_in*C_out. The ratio of separable to standard cost is therefore 1/C_out + 1/k^2 — for a 3x3 kernel into 256 channels that is about 1/256 + 1/9, roughly a 9x reduction in both FLOPs and parameters with little accuracy loss.

This factorization is the engine of mobile and embedded vision. MobileNet (Howard et al., 2017) is built almost entirely from depthwise separable blocks, and Xception (Chollet, 2017) interprets Inception as pushing this separation to the limit, arguing that spatial and cross-channel correlations are sufficiently independent to be learned separately. The same idea generalizes — grouped convolutions are the middle ground between full and depthwise, EfficientNet scales depthwise-separable backbones, and ConvNeXt revisits large-kernel depthwise convolutions to match vision transformers. The trade-off is lower arithmetic intensity: depthwise convolutions are FLOP-light but memory-bandwidth-bound, so the wall-clock speedup on a given accelerator is often less than the FLOP reduction suggests.