semantic segmentation

Semantic segmentation paints every pixel with the name of the category it belongs to, and stops there. If a street scene contains three cars, all of their pixels become the single color "car" — the method does not care that there are three separate vehicles. It answers the question "what kind of thing is at this pixel?" but never "which specific object is this?"

Formally, you choose a fixed set of classes (say sky, road, building, person, car). The network outputs, for each pixel, a probability distribution over those classes; taking the highest-probability class gives the final label map. Training minimizes a per-pixel loss — usually cross-entropy, often combined with a region-overlap loss such as Dice to fight class imbalance. The standard benchmark metric is mean Intersection-over-Union (mIoU): compute IoU between predicted and true masks for each class, then average across classes so that rare classes count as much as common ones.

The architectural lineage runs from fully convolutional networks (the first end-to-end design), through encoder-decoder models like U-Net and SegNet, to context-heavy designs like DeepLab (atrous convolutions, ASPP) and PSPNet (pyramid pooling), and now transformer-based models such as SegFormer and the unified Mask2Former. They all wrestle with the same context-versus-resolution problem: capture enough surrounding scene to label a pixel correctly, while keeping boundaries sharp.