network in network

Network in Network (Lin, Chen, and Yan, 2013) is a short, enormously influential paper whose ideas are now everywhere even though the architecture itself is rarely used by name. Its starting observation is that an ordinary convolution computes, at each spatial location, a linear function of the local patch followed by a single nonlinearity, which is a fairly weak per-location model. NiN proposes replacing that with a tiny multi-layer perceptron applied identically at every location, a 'micro-network' scanning the image. The authors called this an mlpconv layer, and it makes the per-location feature extractor itself nonlinear and more expressive.

The decisive practical realisation is that a multi-layer perceptron applied independently at each spatial position is exactly equivalent to a stack of 1x1 convolutions. A 1x1 convolution does no spatial mixing at all; it simply takes the vector of channel values at one pixel and applies a learned linear map (and then a nonlinearity) to produce a new vector of channels. This is the birth of the 1x1 convolution as a first-class tool for cheaply mixing and projecting channels, increasing or reducing their number, and adding nonlinear capacity without touching spatial resolution. Every later architecture, including the Inception bottleneck, ResNet's bottleneck block, MobileNet's pointwise convolutions, and squeeze-and-excitation, leans on this idea.

NiN's second lasting contribution is global average pooling as a replacement for the final fully-connected classifier. Instead of flattening the last feature maps and running them through large dense layers (the parameter-heavy, overfitting-prone tail of LeNet, AlexNet, and VGG), NiN produces one feature map per class and simply averages each map down to a single number, feeding those directly into the softmax. This has almost no parameters, is far more robust to overfitting, enforces a clean correspondence between feature maps and categories, and accepts variable input sizes. Global average pooling is now standard in essentially every modern CNN, which is why NiN is best understood not as a competitive model but as the source of two components that the whole field absorbed.