mask r-cnn

Mask R-CNN is the canonical detect-then-segment model for instance segmentation, and it is conceptually a very small addition to a detector. Start with Faster R-CNN, which finds objects in two stages: a region proposal network suggests candidate boxes, then a head classifies each box and refines its coordinates. Mask R-CNN, from He, Gkioxari, Dollár, and Girshick (2017), simply bolts on a third, parallel branch that predicts a binary mask inside each detected box. Detection and masking happen together, sharing the same backbone features.

Its most important technical contribution is RoIAlign, which fixes a subtle but damaging flaw in the earlier RoIPool. To extract a fixed-size feature grid from a floating-point box, RoIPool rounded coordinates to integers twice, misaligning features from pixels by up to several pixels — tolerable for a loose box, ruinous for a pixel-accurate mask. RoIAlign removes all rounding and samples feature values at exact floating-point locations using bilinear interpolation, restoring the alignment that crisp masks demand.

Two further design choices matter. The mask branch is a small fully convolutional network applied per region, typically producing a 28×28 mask that is then resized to the box. Crucially, masks are decoupled from classification: the branch predicts one binary mask per class and the classification branch decides which class's mask to keep. This avoids competition between classes and was shown to outperform predicting a single class-aware mask. With a ResNet-FPN backbone (a Feature Pyramid Network handles multiple object scales), Mask R-CNN became the long-standing strong baseline on COCO instance segmentation and extends naturally to human pose by predicting keypoint "masks".