Telling Objects Apart: Instance and Panoptic Segmentation

Why semantic isn't enough

Remember the two parked cars from the very first guide in this track — the silver one half-hidden behind the red one. A model that does semantic segmentation would look at that scene and color every pixel that belongs to any car with the single label 'car'. Because the two cars overlap and touch, those pixels form one connected blob. The network is technically correct — every painted pixel really is car — and yet the answer is useless for the questions we most often want to ask.

Ask 'how many cars are there?' and the semantic map cannot answer — one blob could be one car or five. Ask 'select just the red car so I can erase it' and again you are stuck — there is no boundary inside the blob telling you where one car ends and the next begins. The same failure blocks tracking a single car across video frames. The missing ingredient is instance identity: every individual, countable object should get its own separate mask. That is exactly what instance segmentation delivers — not just what each pixel is, but which specific object it belongs to.

There are two natural strategies for getting instance masks. The first — the focus of this guide — is detect-then-segment: first find each object as a box, then carve out a mask inside that box, so isolating objects one at a time sidesteps the merging problem. The second is the eventual unification: a single output that labels every pixel, giving things their instance ids and stuff its class all at once. That unified view is called panoptic segmentation, and we build up to it at the end.

Detect-then-segment: Mask R-CNN

The cleanest way to understand Mask R-CNN is this: take an object detector you already know — Faster R-CNN — and bolt one extra branch onto it. Faster R-CNN's whole job is to draw a tight box around each object and say what class it is. Mask R-CNN keeps all of that and adds a small network that, for each detected box, paints in the precise shape of the object inside it. The detection gives you where and what; the new branch gives you the exact silhouette.

Let's recap the pipeline at the right altitude. A backbone CNN turns the image into a feature map. A Region Proposal Network (RPN) then slides over that feature map and, using a set of pre-shaped reference boxes called anchors, suggests a few hundred candidate regions that probably contain something. Each proposed region is cropped from the feature map and fed to the heads that make the final decisions.

Faster R-CNN attaches two parallel heads to each proposed region: a classification head that names the object's class (car, person, ...), and a box-regression head that nudges the rough proposal into a tightly fitting box. Mask R-CNN's one structural change is to add a third parallel head: a tiny fully convolutional network (FCN, the same family of pixel-predicting network from guide 2) that outputs a small binary mask — foreground vs. background — inside that single box.

Here is the insight that makes the whole thing work, and that finally beats the 'two cars merge' problem. Because the detector has already isolated one object per box, the mask head no longer faces a crowded scene — it only has to answer a clean, local question: *within this box, which pixels are the object and which are not?* That is just ordinary foreground/background segmentation, the easiest possible version of the task. Two overlapping cars land in two different boxes, each gets its own foreground mask, and so each car emerges as a separate instance — exactly what semantic segmentation could never give us.

RoIAlign: fixing the misalignment

There is one technical innovation without which Mask R-CNN's masks would be blurry and shifted: RoIAlign. To appreciate it, look first at the problem it replaced. To feed a proposed region into the fixed-size heads, the network must crop a fixed-size grid (say 7×7) out of the feature map. The older method, RoIPool, did this crop with two rounding steps.

Here is the trouble. A proposal's coordinates come out as real numbers — a box edge might fall at feature coordinate 6.7, not a clean 7. RoIPool rounds that 6.7 down to 6 (first rounding), then rounds again when it splits the region into the 7×7 sub-cells (second rounding). Each rounding shifts the sampled features by up to a pixel relative to where the object actually is. For a bounding box, a one-pixel slop is invisible — the box still hugs the car. But for a pixel-accurate mask, that same shift smears the silhouette: the predicted outline ends up a pixel off from the true edge everywhere, and the mask looks soft and misregistered.

Mask R-CNN's fix, RoIAlign, is delightfully simple: never round. Keep the region's real-valued coordinates, place the sampling points at their exact fractional positions, and read off the feature value at each such point using bilinear interpolation — blending the four nearest grid cells. The grid alignment is preserved exactly, so the mask lines up with the object pixel-for-pixel.

Bilinear interpolation is just 'reading a value between grid cells by weighted blending'. Imagine your sample point sits inside one unit cell whose four corners hold known feature values. The value you want is a weighted average of those four corners, where a corner gets more weight the closer it is to your point:

Bilinear interpolation: the sampled value is a distance-weighted blend of the four surrounding grid values.

Let's unpack every symbol. The four corners are v_{00} (top-left), v_{10} (top-right), v_{01} (bottom-left) and v_{11} (bottom-right) — the feature values stored at the four grid cells surrounding your sample. The numbers d_x and d_y both live in [0,1] and say how far across the cell your point sits: d_x is the horizontal fraction (0 = flush left, 1 = flush right), d_y the vertical fraction (0 = top, 1 = bottom). Each term multiplies one corner's value by the area of the opposite sub-rectangle — which is exactly why a near corner gets a big weight and a far corner a small one. Concretely, take d_x=0.25, d_y=0.75 with corner values v_{00}=10,\,v_{10}=20,\,v_{01}=30,\,v_{11}=40. The weights are (0.75)(0.25)=0.1875, (0.25)(0.25)=0.0625, (0.75)(0.75)=0.5625, (0.25)(0.75)=0.1875, giving v = 0.1875\cdot10 + 0.0625\cdot20 + 0.5625\cdot30 + 0.1875\cdot40 = 27.5. Notice the answer leans toward 30 and 40 — the bottom corners — because d_y=0.75 put the point near the bottom edge. That smooth, rounding-free read is the whole trick: no jumps, no one-pixel slip, just a clean weighted look-up.

def roi_align_sample(feat, x, y):
    # feat: feature map; (x, y): exact, non-rounded sample location
    x0, y0 = floor(x), floor(y)      # top-left grid cell (integers)
    x1, y1 = x0 + 1, y0 + 1          # neighbours to the right / below
    dx, dy = x - x0, y - y0          # fractional offsets in [0, 1]

    v00, v10 = feat[y0, x0], feat[y0, x1]   # top-left, top-right
    v01, v11 = feat[y1, x0], feat[y1, x1]   # bottom-left, bottom-right

    # weighted blend of the 4 neighbours -- nearer corners weigh more
    return ((1 - dx) * (1 - dy) * v00 + dx * (1 - dy) * v10 +
            (1 - dx) * dy * v01 + dx * dy * v11)

# RoIAlign: sample at exact locations (NO rounding of x, y),
# then average-pool the samples inside each output cell.

Scoring instance masks

Now that the model emits one mask per object, how do we measure whether it is good? We extend the evaluation ideas from guide 2 from whole-image overlap to per-instance overlap. The core tool is mask IoU (Intersection over Union) between one predicted instance mask and one ground-truth instance mask: count the pixels the two masks share, divide by the pixels either mask covers. IoU of 1 means a perfect match; 0 means no overlap at all.

A predicted instance only counts as correct when two conditions both hold: its mask IoU with a ground-truth instance exceeds a chosen threshold (say 0.5), and its predicted class matches that ground truth's class. A beautifully shaped mask labelled 'truck' over a real car does not count; nor does a correctly labelled car whose mask barely overlaps the real one. Both shape and label must agree.

The standard headline metric for instance segmentation is Average Precision (AP), reusing the precision/recall intuition from the detection track. Precision asks 'of the masks I claimed, what fraction were right?'; recall asks 'of the real objects, what fraction did I find?'. As you accept lower-confidence predictions you find more objects (recall up) but make more mistakes (precision down); AP summarizes that whole trade-off curve into one number. Because picking a single IoU threshold is arbitrary, the community reports AP at a loose threshold ([email protected]) and, more tellingly, AP averaged over thresholds from 0.5 to 0.95 — rewarding masks that stay accurate even when you demand near-perfect overlap.

One practical wrinkle: detectors are eager and often emit several overlapping masks for the same object. Non-max suppression (NMS) cleans this up — among a cluster of high-overlap predictions of the same class, keep only the most confident one and discard the rest. Without NMS, AP would be punished for those duplicate false positives.

def nms(masks, scores, iou_thresh=0.5):
    order = argsort(scores)[::-1]   # most confident first
    keep = []
    while order:
        i = order.pop(0)            # take the top-scoring mask
        keep.append(i)
        # drop any remaining mask that overlaps it too much (same object)
        order = [j for j in order if mask_iou(masks[i], masks[j]) <= iou_thresh]
    return keep

Panoptic segmentation: unifying stuff and things

We now have two complementary skills: guide 3 gave us semantic maps that label the amorphous 'stuff' (sky, road) beautifully, and this guide gave us instance masks that separate countable 'things' (each car, each person). Panoptic segmentation fuses them into one coherent output with a strict rule: every pixel gets exactly one semantic label; pixels belonging to a thing also get a unique instance id; pixels belonging to stuff just get their class. No pixel is left unlabelled, and no pixel is claimed by two segments. It is the complete, gap-free, overlap-free picture of a scene.

The practical challenge is reconciliation. The instance branch may hand you overlapping car masks (two cars whose masks share a few border pixels), and the semantic branch may disagree with them at the seams. To produce one non-contradictory labeling, the system must resolve every conflict — typically by resolving overlaps in confidence order and filling the remaining pixels from the semantic 'stuff' map — so that each pixel ends up assigned to exactly one segment. Turning two possibly-disagreeing outputs into one clean map is the real engineering work of panoptic segmentation.

How do we score such a unified output? With a single metric designed for it: Panoptic Quality (PQ). We first match predicted segments to ground-truth segments (a match needs IoU > 0.5, which conveniently guarantees at most one match per segment), then combine the quality and the count of those matches:

Panoptic Quality: total IoU of matched segments, divided by matched plus half-counted unmatched segments.

Let's read it term by term. A true positive (TP) is a matched pair (p,g) — a predicted segment p whose IoU with a ground-truth segment g exceeds 0.5; |TP| is how many such matches there are. A false positive (FP) is a predicted segment that matched nothing (the model hallucinated a segment), and a false negative (FN) is a ground-truth segment the model missed entirely. The numerator \sum_{(p,g)\in TP}\mathrm{IoU}(p,g) adds up the overlap quality of all the matches — so good matches contribute near 1, sloppy ones less. The denominator counts the matches fully but each mistake (a hallucination or a miss) only half — a balanced penalty that punishes both kinds of error without double-counting. A perfect prediction has every segment matched at IoU 1 and no FP/FN, giving PQ = 1.

The elegant part is that PQ factors cleanly into two interpretable pieces — how well you segmented the things you found times how well you found them:

PQ factors as segmentation quality (average IoU of matches) times recognition quality (an F1-like detection score).

The factorization just multiplies and divides the original PQ by |TP|, but it exposes meaning. SQ (segmentation quality) is the average IoU over only the matched segments — it asks 'when you did find an object, how tightly did your mask hug it?'. RQ (recognition quality) is exactly the F1 score of detection — |TP| over matches-plus-half-the-mistakes — it asks 'did you find the right set of objects, without inventing or missing any?'. A model can score low two very different ways: sloppy masks (low SQ) or right shapes but wrong object counts (low RQ), and PQ tells you which. Worked example: 3 real cars; the model nails 2 (IoU 0.9 and 0.8), invents 1 extra (FP = 1), and misses 1 (FN = 1). Then SQ = (0.9+0.8)/2 = 0.85, RQ = 2 / (2 + 0.5 + 0.5) = 0.667, and PQ = 0.85 × 0.667 ≈ 0.567 — which matches plugging straight into the first formula: 1.7 / 3 ≈ 0.567. The masks it drew were good (SQ high), but it got the object set wrong (RQ dragged it down).

Take the street scene one last time. The panoptic output paints the road and sky as single stuff regions, gives each car and each pedestrian its own crisp, separately-colored mask, and leaves not a single pixel ambiguous. That one image carries everything: what is where (semantic), which individual is which (instance), all reconciled into a unified, contradiction-free description of the scene — the goal the whole track has been building toward.

Where fixed-category models fall short

Step back and notice what every model in this track shares — from the first semantic FCN to Mask R-CNN to panoptic systems. Each is trained on a fixed list of categories and a fixed task. The network can only ever output one of the classes it was trained on. Want it to segment a new kind of object — a specific tool, an animal it never saw, a logo? You must gather labelled data for that class and retrain. And there is no way to simply point at an arbitrary thing in the image — 'segment this, whatever it is' — and have the model carve it out. Its vocabulary, and its very notion of what counts as an object, are frozen at training time.

This is exactly the limitation the final guide breaks open. The next shift is toward promptable, open-world, foundation-model segmentation: a model you can guide at inference time with a hint — a click, a box, a scribble — to segment any object, including ones from categories it was never explicitly taught. Notice that the seed of this idea is old: GrabCut, all the way back in guide 1, already let you 'give the model a hint' — a rough rectangle around the thing you wanted — and refine a mask from it. The next guide scales that humble hint-driven idea up with the power of modern foundation models, so that one general model can segment almost anything you point at.