instance segmentation

Instance segmentation is semantic segmentation with identities. It not only says "these pixels are car" but also "this car is car #1, that one is car #2, the third is car #3", giving each individual object its own pixel-precise mask. In effect it fuses object detection (find and box each thing) with mask prediction (outline it exactly). It is what you need whenever counting or tracking distinct objects matters — cells in a microscope image, people in a crowd, fruit on a conveyor.

The dominant designs come in three families. Detect-then-segment (two-stage), exemplified by Mask R-CNN: first propose object boxes, then predict a binary mask inside each box. Single-stage methods like YOLACT, SOLO, and SOLOv2 skip explicit proposals and predict masks directly, trading some accuracy for speed. Query-based transformer methods like MaskFormer and Mask2Former treat segmentation as a set-prediction problem (in the spirit of DETR): a fixed number of learned queries each emit one mask plus a class, and bipartite matching assigns them to ground-truth objects. This last family is notable because the same model can do semantic, instance, and panoptic segmentation.

Evaluation differs sharply from semantic segmentation. Because outputs are a set of separate, possibly overlapping instance masks each with a confidence score, the metric is mask Average Precision (AP), computed by sweeping a confidence threshold and a mask-IoU threshold (often averaged over IoU from 0.5 to 0.95, the COCO protocol). A prediction counts as correct only if it both has the right class and overlaps a true instance above the IoU threshold.