intersection over union

Intersection over union measures how much two boxes overlap, on a scale from 0 (no overlap) to 1 (identical). The recipe is in the name: take the area where the two rectangles overlap (intersection), divide by the total area they jointly cover (union). If a predicted box sits exactly on the true box, IoU = 1; if they barely touch, IoU is near 0. It is the single most important geometric quantity in detection.

Why divide by the union rather than, say, by the smaller box? Because the union penalizes both kinds of error at once: a box that is too small misses part of the object (shrinking the intersection), and a box that is too large covers empty space (inflating the union). Both drive IoU down, so a high IoU certifies a box that is tight, not merely overlapping. Formally, for boxes A and B, IoU = area(A ∩ B) / area(A ∪ B) = area(A ∩ B) / (area(A) + area(B) − area(A ∩ B)). This is exactly the Jaccard index of the two pixel sets.

IoU does three jobs in a detector. (1) Label assignment: it decides which anchors/proposals are positives for which object. (2) Non-maximum suppression: it decides when two detections are 'the same object' and one should be dropped. (3) Evaluation: a detection counts as correct only if its IoU with a true box clears a threshold — COCO averages mAP over thresholds 0.50 to 0.95 in steps of 0.05, rewarding boxes that are not just present but precise.

Plain IoU has a blind spot as a loss: if two boxes don't overlap at all, IoU is 0 and its gradient is zero everywhere, giving the optimizer no signal about which direction to move. Generalized IoU (GIoU) fixes this by also accounting for the smallest box enclosing both, so non-overlapping boxes still produce a useful gradient; DIoU and CIoU further add center-distance and aspect-ratio terms for faster, more stable convergence.