encoder-decoder architecture

The encoder-decoder is the workhorse blueprint for dense prediction, and the cleanest answer to the context-versus-resolution dilemma. It has two halves that mirror each other. The encoder (the contracting path) repeatedly downsamples the image, shrinking the spatial size while growing the number of feature channels; as it does so each neuron sees a larger and larger patch of the original image, building up abstract, semantic understanding — "this region is fur", "this is part of a wheel". The decoder (the expanding path) then reverses the process, upsampling step by step to rebuild a full-resolution prediction.

The reason this two-stage shape is necessary is that the encoder, while gaining semantics, has thrown away precise location. The decoder's job is to take "I know this is a cat" (from the bottleneck, where the representation is smallest and most abstract) and turn it back into "and the cat's boundary runs exactly here". To do that well, almost every encoder-decoder adds skip connections that pipe high-resolution feature maps from encoder layers directly across to the matching decoder layers, reinjecting the fine spatial detail the bottleneck lost.

Concrete instances reveal the design space. U-Net concatenates the full encoder feature maps into the decoder (rich but memory-hungry). SegNet instead stores only the pooling indices from the encoder and uses them to place values during unpooling (memory-light). DeepLabv3+ uses a strong atrous-convolution encoder and a deliberately simple decoder. The encoder is frequently a pretrained classification backbone — ResNet, EfficientNet, or a Vision Transformer — which transfers general visual knowledge into the segmentation task.