panoptic segmentation

Panoptic segmentation is the unification of the two worlds. The insight is that an image contains two kinds of content. "Things" are countable objects with clear shape — people, cars, dogs — for which you want individual instances. "Stuff" is amorphous, uncountable background — sky, road, grass, water — for which counting is meaningless and only the class matters. Panoptic segmentation labels things with instance IDs and stuff with semantic classes, in one single, complete, non-overlapping map. The name evokes "panopticon": one view that sees everything.

The defining constraint is what makes it coherent: every pixel is assigned exactly one pair (class, instance ID), with no gaps and no overlaps. This is stricter than instance segmentation, where masks may overlap and many pixels may be left unlabeled, and richer than semantic segmentation, which has no instance IDs at all. Resolving the inevitable conflicts (two predicted masks claiming the same pixel) is part of the task.

Performance is summarized by Panoptic Quality (PQ), introduced with the task by Kirillov and colleagues in 2019. For each class, match predicted to true segments using IoU > 0.5 (which guarantees a unique match), then PQ = (sum of IoU over matched true positives) divided by (TP + 0.5·FP + 0.5·FN). This factors neatly into PQ = SQ × RQ, where Segmentation Quality is the average IoU of matched segments (how well-shaped the matches are) and Recognition Quality is an F1-like detection score (how many segments were found at all). PQ is then averaged over classes.