semantic segmentation

Semantic segmentation is the vision task of coloring in every single pixel of an image with a label that says what kind of thing it belongs to. Instead of drawing a loose box around a car, the system decides, for each tiny dot in the picture, whether it is road, sky, car, tree, or person — producing a kind of paint-by-numbers map laid exactly over the photo. Where object detection gives you a rough rectangle, segmentation traces the true silhouette of everything, right down to the jagged edge of a leaf or the curve of a curb.

The word 'semantic' just means the labels are about meaning, the categories of things, not individual identities. So in a street scene, all the cars get painted the same 'car' color and all the pedestrians the same 'person' color; semantic segmentation does not bother to separate car number one from car number two. (Telling apart individual instances is a related but distinct task called instance segmentation.) This pixel-perfect map of categories is exactly what a self-driving car wants when it needs to know precisely where the drivable road ends and the sidewalk begins.

Like other modern vision tasks, semantic segmentation is done with neural networks trained on images where humans have painstakingly hand-labeled every pixel. The network takes the whole image in and outputs a label for each pixel at once. For robots, this dense understanding feeds navigation (what is ground I can roll on?), manipulation (where exactly is the surface of this object?), and safe operation around people and obstacles.