anchor box

An anchor box is a guess made before looking at the image — a predefined rectangle of a chosen size and shape, placed at a grid location, that the detector will fine-tune into an actual prediction. Instead of asking the network to conjure a box out of nothing (a hard, unconstrained regression), we hand it a menu of starting rectangles and ask only 'nudge this one to fit, and tell me if there's an object in it'. Anchors turn an open-ended search into many small, easy corrections.

At every spatial cell of a feature map, a detector tiles several anchors that differ in scale (small, medium, large) and aspect ratio (tall, square, wide), because objects come in many shapes. A typical setup is 3 scales × 3 ratios = 9 anchors per cell; over a whole feature map this yields tens of thousands of anchors covering the image densely. Each anchor is then responsible for predicting (a) an objectness/class score and (b) four offsets that warp the anchor into a tight box.

During training, every anchor is assigned a label by how well it overlaps the ground-truth boxes via IoU: anchors with high overlap (e.g. IoU ≥ 0.7) become positives tied to a specific object, those with low overlap (e.g. IoU < 0.3) become negatives (background), and the in-between ones are ignored so they don't send confusing gradients. Positives also get a regression target — the offset that moves the anchor onto its object. This is 'label assignment', and its rules quietly determine much of a detector's accuracy.

Anchors are powerful but introduce hyperparameters that must roughly match your data: the scales and ratios, the IoU thresholds, and how anchors are spread across feature-map levels. Pick them badly — say, no tall thin anchors when your dataset is full of pedestrians — and recall collapses. This brittleness is exactly what later anchor-free detectors (FCOS, CenterNet) and set-prediction methods (DETR) set out to remove.