Teaching a Network to Color Every Pixel: FCN, Encoder–Decoder, and U-Net

The dense prediction problem

In the first guide of this track we redrew the goal of image segmentation: instead of one label for a whole image, we want a label for every single pixel. A street photo becomes a colour-by-number map where each pixel is tagged 'road', 'car', 'person', or 'sky'. This per-pixel labelling is semantic segmentation, and it is an example of what the field calls dense prediction — we must output something dense, one answer per input location, not a single summary verdict.

To see why this is hard, recall how an ordinary classification CNN works — the kind from the CNN track that answers 'cat or dog?'. The image flows through stacks of convolution + pooling layers. Each convolution detects patterns; each pooling step throws away half the width and half the height to stay cheap and to let later layers see more of the image at once. After several stages a 224×224 image has shrunk to a tiny feature grid — perhaps 7×7 — with many channels. Finally, fully connected layers flatten that grid into one long vector and squash it to a single list of class scores: one number per category.

That design is brilliant for classification and disastrous for segmentation, because two things were destroyed along the way. First, pooling threw away spatial resolution: a 7×7 grid simply cannot say which of the original ~50,000 pixels is a car and which is road. Second, the fully connected layers threw away the spatial grid entirely — once you flatten a grid into a vector, the network no longer knows that 'this score came from the top-left' and 'that one from the bottom-right'. Geometry is gone. We are left with what is in the image but no longer where.

Fully Convolutional Networks

The first answer, from 2015, is the fully convolutional network (FCN), and its name is a promise: no fully connected layers at all — every layer is a convolution. The first move is a neat trick. A dense layer that maps a 7×7×512 grid to 4096 numbers is mathematically identical to one big convolution with a 7×7 kernel, so we can drop the dense classifier and use 1×1 convolutions that emit, at every location of the feature grid, one score per class. The payoff is twofold: the output becomes a coarse grid of class scores — a low-resolution heatmap per class (a 7×7×C block, where the 'car' channel lights up over cars and the 'road' channel over road) — and, with no fixed-size dense layers, the network now accepts any input size; a bigger image simply yields a bigger heatmap.

We now hold a 7×7 heatmap but need a 224×224 label map. That is the second move: upsampling, done with a transposed convolution (sometimes called a deconvolution — a slightly misleading name). Picture it as the reverse of striding. A strided convolution skips over the input to make the output smaller; a transposed convolution spreads the inputs apart to make the output bigger. Concretely, each input cell stamps a scaled copy of a learned kernel onto a larger output grid, and where stamps overlap, their values are summed. The kernel is learned, so the network discovers how to fill the gaps instead of obeying a fixed rule like nearest-neighbour.

Output size of a transposed convolution.

Read this as: the output side length O is grown from the input, not shrunk. I is the input spatial size (one side, in cells), s is the stride — here the upsampling factor — p is the padding, and k is the kernel size. Substitute a tiny case: I = 2, s = 2, p = 0, k = 2 gives O = (2−1)·2 − 0 + 2 = 4, so the map doubled from 2 to 4 — exactly what a stride-2 upsample should do. Contrast the ordinary-convolution shrinking formula you already met, O = ⌊(I − k + 2p)/s⌋ + 1: with I = 4, k = 2, s = 2 it gives 2, going 4 → 2. They are mirror images — the same stride that made a normal conv skip and shrink now spreads inputs apart and grows. (FCN's real map needs 32× upsampling, because five poolings each halved it: 2⁵ = 32.) One catch remains: a single 32× jump is hopelessly blurry, since the fine detail was destroyed back at the pooling stages. FCN's remedy is the skip connection — it upsamples partway, then adds in a sharper class-score prediction taken from an earlier, higher-resolution layer (the 1/16 and 1/8 maps), then upsamples again. The result (FCN-8s) has visibly crisper edges, and this 'combine deep-but-coarse with shallow-but-fine' idea seeds the next two sections.

The encoder–decoder idea

FCN's two halves — shrink, then grow back — crystallize into the pattern that dominates segmentation today: the encoder–decoder architecture (you may also hear 'contracting and expanding paths'). The encoder is the contracting path: it repeatedly downsamples with convolution and pooling, just like a classification backbone, distilling the image into ever-smaller, ever-deeper feature maps.

Why downsample at all? Because it grows the receptive field — the patch of the original image a single deep feature can 'see'. Early on, a feature reacts to a tiny edge; after several poolings, one feature summarizes a large region and can recognize 'this is part of a dog'. Downsampling trades resolution for context: the network increasingly understands what is present while progressively forgetting where it is. The intuition is squinting — half-close your eyes and a cluttered scene resolves into its gist ('kitchen', 'street') even though you can no longer read the fine detail.

At the deepest point sits the bottleneck: the smallest, most compressed, most semantic representation — rich in 'what', nearly empty of 'where'. The decoder (the expanding path) now makes the reverse journey. It repeatedly upsamples — with transposed convolutions or simpler interpolation — step by step rebuilding spatial resolution until the map is back at full size, one class score per original pixel. If the encoder squints to grasp the gist, the decoder opens its eyes again to place every boundary precisely.

U-Net: skip connections that save the details

U-Net, introduced in 2015 for biomedical images, is the encoder–decoder architecture refined to near-perfection — and it remains a default choice a decade later. Drawn out, it looks like a symmetric letter U: the encoder descends the left arm, downsampling, and the decoder climbs the right arm, upsampling, with the bottleneck at the bottom. What makes it special is the set of skip connections that bridge straight across the U, from each encoder level to the decoder level at the same resolution.

Here is the key trick, and it is delightfully simple. At each decoder level, before continuing upward, U-Net takes the matching high-resolution feature map saved from the encoder and concatenates it onto the upsampled decoder feature map — literally stacking them along the channel axis — then applies a couple of convolutions to fuse the two. So the decoder never has to hallucinate the fine boundary detail it lost during downsampling: that detail is handed back directly from the encoder, at exactly the right resolution, right when it is needed.

Why does this work so well, especially for crisp edges? Boundaries are high-frequency information — they live in the earliest, highest-resolution feature maps and are the very first thing pooling smears away. By piping those early maps straight to the decoder, U-Net lets the final layers sharpen object outlines using genuine fine-grained evidence, not a blurry guess. This is also why it took over medical imaging: clinical datasets are often small (dozens of annotated scans, not millions), and the skip connections make U-Net remarkably data-efficient, while precise boundaries matter enormously when you are outlining an organ or a tumor for treatment.

One subtle contrast with FCN is worth naming. FCN's skip connections add the encoder's prediction into the decoder (element-wise sum); U-Net's skip connections concatenate the encoder's features (stack channels) and let the following convolutions decide how to combine them. Addition is cheaper and keeps the channel count fixed, but it forces both sources into the same 'slots'. Concatenation keeps both streams intact and gives the network freedom to learn the best fusion — at the cost of more channels (and a bit more compute). U-Net's choice of concatenation is a big part of why its boundaries look so clean.

# One U-Net decoder step: upsample, then CONCATENATE the
# matching encoder feature map before convolving to fuse.
def decoder_block(x, skip, up, conv):
    x = up(x)                        # transposed conv: double H and W
    x = torch.cat([x, skip], dim=1)  # glue channels: decoder + encoder detail
    x = conv(x)                      # fuse the two streams with normal convs
    return x

# FCN would instead ADD a prediction:  x = up(x) + skip_pred

Training: the loss that scores a mask

How do we actually train one of these networks? The simplest and most common loss is per-pixel cross-entropy. The idea is disarmingly direct: treat every single pixel as its own little classification problem — 'is this pixel road, car, or sky?' — compute the ordinary classification loss there, and then average that loss over all the pixels. If the predicted label map is right everywhere, the loss is near zero; where it is confidently wrong, the loss is large.

Per-pixel (mean) cross-entropy loss.

Symbol by symbol: N is the number of pixels, C the number of classes. y_{i,c} is the ground truth, written one-hot — it is 1 if pixel i's true class is c, and 0 otherwise. ŷ_{i,c} is the network's predicted probability (after a softmax) that pixel i is class c. Because y is one-hot, the inner sum keeps only the log-probability assigned to the correct class; the minus sign turns 'high probability = good' into 'low loss = good', and 1/N averages over the image. Now the weakness: suppose you are segmenting a tiny tumor that occupies 1% of the pixels. A lazy network can predict 'background' for every pixel and instantly be 99% accurate with a tiny cross-entropy — yet it found nothing. This is the class imbalance trap, and it bites hardest exactly where segmentation matters most: thin road lines, small lesions, distant pedestrians. The cure is to stop rewarding per-pixel correctness and start rewarding overlap with the target — which is what the Dice coefficient measures.

Dice coefficient (an overlap score) and the Dice loss.

Read the Dice coefficient as twice the shared area, divided by the total area claimed by both sides. P is the set of pixels the model predicts as the class; G is the set of ground-truth pixels of that class; |·| means 'count the pixels'; and |P ∩ G| is the intersection — the pixels both sides agree on. Why the factor of 2? Without it, even a perfect prediction (P = G) would give |P|/(|P|+|P|) = 0.5; the 2 rescales a perfect match to 1, while no overlap at all gives 0. Make it concrete on a 2×2 image where 1 marks the class. Ground truth G = [[1,1],[0,0]] so |G| = 2; prediction P = [[1,1],[1,0]] so |P| = 3; they agree on the two top pixels, so |P ∩ G| = 2. Then D = 2·2 / (3 + 2) = 4/5 = 0.8. To turn a score (higher is better) into a loss (lower is better) we simply define Dice loss = 1 − D = 0.2 and minimize it by gradient descent. Because both numerator and denominator scale with the object, a huge background no longer drowns out a tiny target — exactly the imbalance fix we wanted.

import numpy as np

# One class on a 2x2 image; 1 = "this pixel is the class"
G = np.array([[1, 1],
              [0, 0]])              # ground truth, |G| = 2
P = np.array([[1, 1],
              [1, 0]])              # prediction,   |P| = 3

inter = np.sum((P == 1) & (G == 1))      # pixels both agree -> 2
dice  = 2 * inter / (P.sum() + G.sum())  # 2*2 / (3 + 2) = 0.8
dice_loss = 1 - dice                     # 0.2  (this is what we minimize)
print(dice, dice_loss)                   # 0.8 0.2

Measuring success: IoU and friends

Training is only half the story; we also need to measure success honestly, and the same imbalance trap haunts evaluation. Plain pixel accuracy — the fraction of pixels labelled correctly — is misleading for exactly the reason cross-entropy was: that 99%-accurate tumor detector found nothing. The standard fix is Intersection-over-Union (IoU), also called the Jaccard index. For one class it asks: of all the pixels that either the prediction or the ground truth calls this class, what fraction did they agree on?

Intersection-over-Union (Jaccard index) for one class.

Same P and G as before: P ∩ G is the agreement (2 pixels), and P ∪ G is the union — every pixel predicted or labelled as the class, each counted once. In our 2×2 example P (3 pixels) and G (2 pixels) together cover three distinct cells, so |P ∪ G| = 3 and IoU = 2/3 ≈ 0.667. Compare the two denominators: Dice divides by |P| + |G| (which double-counts the overlap), whereas IoU divides by the union (which counts it once). Because IoU's denominator is effectively larger, IoU is always the stricter, smaller number — in fact Dice ≥ IoU always, and here 0.8 ≥ 0.667. They are cousins: both pure overlap measures, linked by D = 2·IoU/(1+IoU). Dice is popular as a loss; IoU is the conventional report metric.

Mean IoU averaged across all C classes.

A scene has many classes, so we compute IoU per class and then average: IoU_c is the IoU for class c, C is the number of classes, and mIoU — mean IoU — is their simple average. This per-class averaging is the whole point. If we instead pooled all pixels together, the 'sky' and 'road' classes, with their millions of pixels, would dominate the score and a model could ignore 'pedestrian' entirely without much penalty. By giving every class one equal vote, mIoU forces the network to do well on the rare, small classes too — which is precisely why mIoU is the headline number reported on benchmarks like Cityscapes and Pascal VOC. With these metrics in hand, the next guide can push the what-vs-where fight further: how to give the encoder a huge field of view without destroying the resolution the decoder needs.