densenet

DenseNet (Huang, Liu, van der Maaten, and Weinberger, 2017) takes the residual-connection insight, that shortcut paths help gradients and features flow, and pushes it to an extreme. Whereas a ResNet block adds its input to its output, a DenseNet connects every layer to every other layer that follows it within a block, and combines them by concatenation rather than addition. Concretely, layer number k receives, as its input, the concatenated feature maps of all preceding layers in the block (the original input plus the outputs of layers 1 through k-1), and its own output is then made available to every later layer. In a block of L layers there are L(L+1)/2 direct connections instead of just L.

The practical payoffs are feature reuse and strong gradient flow. Because every layer can directly see the raw features produced by all earlier layers, the network does not have to relearn and re-transmit information through the depth, so each layer can be very thin, adding only a small fixed number of new feature maps called the growth rate (typically 12, 24, or 32). This makes DenseNets remarkably parameter-efficient: they often match ResNet accuracy with substantially fewer parameters. And because every layer has a short path to the loss and to the input, gradients propagate well and the network has a built-in form of deep supervision.

A DenseNet is built from dense blocks separated by transition layers. Concatenation only works when feature maps share spatial dimensions, so within a dense block resolution is held constant; the transition layers between blocks apply a 1x1 convolution (to compress the accumulated channels) and a 2x2 average pooling (to downsample) before the next block. The main trade-off is memory: although DenseNet has few parameters, naively storing all the concatenated activations for backpropagation consumes a lot of memory, which led to memory-efficient implementations that recompute concatenations on the backward pass. DenseNet remains a clean illustration that how you connect layers matters as much as how many you stack, and its feature-reuse philosophy contrasts instructively with ResNet's additive shortcuts.