feature pyramid network

A feature pyramid network (Lin et al., 2017) is a neck — a module inserted between the backbone and the detection heads — that builds a set of feature maps at multiple scales where every level is both high-resolution enough to localize and semantically rich enough to classify. The problem it solves: a plain CNN's deep layers have powerful semantics but coarse resolution (bad for small objects), while its shallow layers have fine resolution but weak semantics (bad for recognition). FPN gives you the best of both at every scale.

The trick is a top-down pathway with lateral connections. The backbone produces a bottom-up pyramid C2…C5 (progressively deeper, smaller, more semantic). FPN starts from the deepest map, then repeatedly upsamples it 2× and adds it (element-wise, after a 1×1 conv that matches channel counts) to the corresponding bottom-up map — injecting strong semantics into higher-resolution levels. A 3×3 conv smooths each merged map to remove upsampling artifacts, yielding output maps P2…P5 (and often P6/P7) that are all semantically strong.

Detection then runs the same head on every pyramid level, with each level handling a band of object sizes: large objects are detected on coarse, deep levels and small objects on fine, shallow-but-now-semantic levels. In Faster R-CNN with FPN, a proposal is routed to the pyramid level matching its size; in one-stage detectors like RetinaNet, anchors of different scales are assigned to different levels. The compute overhead is small because the top-down path is cheap relative to the backbone.

FPN became a near-universal building block: it boosts small-object accuracy across detectors (Faster R-CNN, RetinaNet, Mask R-CNN, most YOLO necks) and underlies later variants like PANet (which adds a bottom-up path on top of the top-down one), BiFPN (learned weighted, repeated fusion in EfficientDet), and NAS-FPN (searched topologies). If you see a modern detector handle objects from tiny to huge well, there is almost always a feature pyramid inside.