Scoring Detections: IoU, NMS, and mAP

How much do two boxes overlap? Intersection over Union

In the previous guide we said a detector outputs a list of bounding boxes, each with a class and a confidence. But how do we tell whether a predicted box is actually on the object, or just floating nearby? We need a single number that measures how well two rectangles agree. That number is the intersection over union, almost always written IoU. It is the universal yardstick of object detection: matching predictions to ground truth, removing duplicates, and scoring a model all rest on it.

Here is the intuition before any algebra. Imagine two clear plastic boxes laid on a table. Their intersection is the patch of table covered by both boxes at once. Their union is the patch covered by at least one of them. IoU is simply the overlap divided by the total covered area. If the boxes do not touch, the overlap is zero, so IoU is 0. If they sit exactly on top of each other, the overlap equals the union, so IoU is 1. Everything useful lives in between. A lovely property falls out for free: because it is a ratio of two areas, IoU is scale-invariant — doubling the size of both boxes leaves the value unchanged, so the same threshold works for a tiny bird and a huge bus.

Intersection over union for two boxes A and B.

Reading it symbol by symbol: A and B are the two boxes (say, a prediction and a ground-truth box). A \cap B is their intersection — the region inside both — and |A \cap B| is its area (the bars mean 'area of'). A \cup B is their union — the region inside either — and |A \cup B| is its area. We divide overlap by total coverage. To actually compute it we need the corners. Suppose each box is stored as (x_1, y_1, x_2, y_2): the top-left corner (x_1,y_1) and the bottom-right corner (x_2,y_2), with x growing rightward and y growing downward (the usual image convention). Then:

Building the intersection rectangle from corners, then the union.

Walk through every line. The overlap rectangle's left edge is the rightmost of the two left edges, x_{\text{left}}=\max(x_1^A, x_1^B) — overlap can only start where both boxes have already begun. Its right edge is the leftmost of the two right edges, x_{\text{right}}=\min(x_2^A, x_2^B) — overlap must end as soon as either box ends. The top and bottom edges follow the same logic vertically. The overlap width is x_{\text{right}}-x_{\text{left}} and its height is y_{\text{bottom}}-y_{\text{top}}; multiply them for the area \text{inter}. The crucial guard is \max(0, \cdot): if the boxes do not overlap along an axis, that difference comes out negative, and a negative width times a positive height would give a nonsensical negative 'area'. Clamping to zero makes a non-overlapping pair score exactly \text{inter}=0, hence \mathrm{IoU}=0, as it must. Finally, the union is the two box areas added together minus the intersection — because when you add the areas, the overlapping patch got counted twice (once in each box), so we subtract it once to avoid double-counting.

def iou(boxA, boxB):
    # boxes are (x1, y1, x2, y2): top-left and bottom-right corners
    x_left   = max(boxA[0], boxB[0])
    y_top    = max(boxA[1], boxB[1])
    x_right  = min(boxA[2], boxB[2])
    y_bottom = min(boxA[3], boxB[3])

    # clamp to 0: if the boxes do not overlap, width or height goes negative
    inter_w = max(0, x_right - x_left)
    inter_h = max(0, y_bottom - y_top)
    inter   = inter_w * inter_h

    areaA = (boxA[2] - boxA[0]) * (boxA[3] - boxA[1])
    areaB = (boxB[2] - boxB[0]) * (boxB[3] - boxB[1])
    union = areaA + areaB - inter        # subtract inter so it is not counted twice

    return inter / union if union > 0 else 0.0

# Worked example
A = (1, 1, 5, 4)   # areaA = 4 * 3 = 12
B = (3, 2, 7, 6)   # areaB = 4 * 4 = 16
# x_left=3, y_top=2, x_right=5, y_bottom=4 -> inter = 2 * 2 = 4
# union = 12 + 16 - 4 = 24  ->  IoU = 4 / 24 = 0.167
print(round(iou(A, B), 3))   # 0.167

C = (10, 10, 12, 12)         # far away from A: no overlap
# x_right - x_left = 5 - 10 = -5  ->  max(0, -5) = 0  ->  inter = 0
print(iou(A, C))             # 0.0

Let us read the worked numbers, because this is the calculation you will redo a thousand times. Box A=(1,1,5,4) has area 4\times3=12; box B=(3,2,7,6) has area 4\times4=16. The intersection corners are x_{\text{left}}=\max(1,3)=3, y_{\text{top}}=\max(1,2)=2, x_{\text{right}}=\min(5,7)=5, y_{\text{bottom}}=\min(4,6)=4, giving an overlap of 2\times2=4. The union is 12+16-4=24, so \mathrm{IoU}=4/24\approx0.167 — the boxes share only a sixth of their combined footprint. Now box C=(10,10,12,12) sits far from A: x_{\text{right}}-x_{\text{left}}=5-10=-5, clamped to 0, so the intersection is 0 and \mathrm{IoU}=0. That one number — matching, suppression, and evaluation all flow from it.

Deciding hit or miss: IoU thresholds

A box that overlaps the truth a little is better than one that misses entirely, but to score a detector we eventually need a yes/no verdict on each prediction. The IoU turns the fuzzy question 'is this box right?' into a crisp rule: pick a threshold (0.5 is the classic choice) and say a prediction is geometrically good enough only if its IoU with a ground-truth box clears that bar. Combined with the predicted class, this lets us sort every prediction in object detection into exactly one of three buckets.

The three buckets are: a true positive (TP) is a prediction of the correct class whose IoU with a ground-truth box exceeds the threshold and which is the first prediction to claim that particular object. A false positive (FP) is a prediction that matches nothing — it has low IoU with every ground-truth box, has the wrong class, or it is a duplicate of an object some higher-confidence box already claimed. A false negative (FN) is a real object that no prediction ever matched — something the detector simply missed. (Detection has no useful 'true negative': the empty background is not an object we count.)

1. Sort all predictions of one class by confidence, highest first, then go down the list.
2. For the current prediction, find the still-unmatched ground-truth box of the same class with the largest IoU.
3. If that best IoU exceeds the threshold (say 0.5), mark this prediction a true positive and lock that ground-truth box as taken.
4. Otherwise — no box clears the bar, or every good box is already taken — mark the prediction a false positive.
5. After all predictions are processed, every ground-truth box still unmatched becomes a false negative.

A tiny example makes it concrete. Suppose an image has two real dogs, G_1 and G_2, and the detector outputs three boxes: P_1 (conf 0.9, IoU 0.8 with G_1), P_2 (conf 0.8, IoU 0.6 with G_1), P_3 (conf 0.7, IoU 0.1 with everything). Sorted by confidence we take P_1 first: 0.8 beats 0.5, so P_1 is a TP and G_1 is now taken. Next P_2: its best overlap is with G_1, but G_1 is already claimed, so P_2 is a duplicate FP. Then P_3: no box clears 0.5, so it is an FP too. Finally G_2 was never matched, so it is an FN. Tally: 1 TP, 2 FP, 1 FN — a detector that found one of two dogs and emitted two pieces of junk. Those four counts are the raw material for every metric that follows.

Removing duplicates: Non-Maximum Suppression

Guide 1 promised that detectors emit a tangle of overlapping boxes and that we have a cleanup step for it. Here it is. Modern detectors are dense: they test thousands of candidate locations, and a confident object lights up not one but a whole cluster of nearby bounding boxes, all wrapping the same dog with slightly different edges. The previous section showed why that hurts — every box past the first becomes a false positive. We want to keep the single best box per object and delete the rest. The standard tool is non-maximum suppression (NMS).

1. Take all candidate boxes of one class and sort them by confidence score, highest first.
2. Pop the top box, call it M, and add it to the kept list — it is the local winner.
3. Compute IoU(M, b) for every remaining box b, and discard any b whose IoU exceeds the NMS threshold — these are M's duplicates.
4. Repeat on whatever boxes survive, until none are left. The kept list is the final detection set.

The core suppression rule of greedy NMS.

Symbol by symbol: M is the highest-scoring box still in the running at this step — the current 'maximum', which is exactly why the method is named non-maximum suppression: we keep the maximum and suppress the others around it. b_i ranges over every other candidate box that has not yet been kept or removed. N_t is the NMS IoU threshold, the overlap level above which we declare b_i a duplicate of M and delete it. The rule reads: accept M, then erase any b_i that overlaps M by at least N_t. Why does the loop terminate and leave one box per object? Each pass permanently removes M (into the kept list) plus all its near-twins from the pool, so the candidate set strictly shrinks every round and must hit empty. And because all the boxes piled on one object overlap heavily with that object's winner, they get wiped in the round that winner is chosen — leaving exactly one survivor per cluster.

def nms(boxes, scores, iou_threshold):
    # boxes: list of (x1,y1,x2,y2); scores: matching confidence per box
    order = sorted(range(len(boxes)), key=lambda i: scores[i], reverse=True)
    keep = []
    while order:
        m = order.pop(0)          # M = highest-scoring box still in the running
        keep.append(m)
        # drop every remaining box that overlaps M too much (a duplicate)
        order = [i for i in order if iou(boxes[m], boxes[i]) < iou_threshold]
    return keep

Trace a four-box example with N_t=0.5. Suppose B_1 (score 0.95), B_2 (0.90), B_3 (0.85) all hug the same dog, while B_4 (0.80) sits on a second dog across the image. The overlaps are \mathrm{IoU}(B_1,B_2)=0.82, \mathrm{IoU}(B_1,B_3)=0.74, \mathrm{IoU}(B_1,B_4)=0.05. Round one: M=B_1 (the maximum), accept it; B_2 and B_3 both exceed 0.5 so they are deleted as duplicates, while B_4 at 0.05 is well below 0.5 and survives. Round two: only B_4 remains, so M=B_4, accept it; nothing left to compare. Result: \{B_1, B_4\} — one clean box per dog, two false positives removed. That is exactly the cleanup guide 1 promised.

Precision and recall for detectors

Once NMS has cleaned up the boxes and we have tallied TP, FP, and FN, two complementary questions remain about a detector. First: of the boxes it did predict, how many were actually right? Second: of the real objects out there, how many did it find? These are precision and recall, the two halves of how we judge object detection quality. A detector can be great at one and terrible at the other, so we always look at both together.

Precision and recall from the detection counts.

Read both fractions with the buckets from section 2. TP is the number of correct boxes (right class, IoU over threshold, not a duplicate); FP is the number of wrong or duplicate boxes; FN is the number of real objects missed entirely. Precision =TP/(TP+FP) divides correct predictions by all predictions made — its denominator is everything the detector emitted, so it answers 'when it speaks up, how often is it right?' Recall =TP/(TP+FN) divides correct predictions by all real objects — its denominator is the full set of ground-truth objects, so it answers 'of everything that was there, how much did it catch?' Take our earlier tally of 1 TP, 2 FP, 1 FN: precision =1/(1+2)=0.33 and recall =1/(1+1)=0.50 — it caught half the dogs but two-thirds of its boxes were junk.

Here is the key dynamic that the next metric is built on: precision and recall trade off as we move the confidence threshold. Every box carries a confidence, and we get to choose how confident a box must be before we keep it. Lower the threshold and the detector reports more boxes — it finds more of the real objects (recall rises) but also lets through more junk (precision falls). Raise the threshold and it only reports its surest boxes — those are usually right (precision rises) but it stays silent on the harder objects (recall falls). Sweeping the threshold from strict to lenient drags the operating point along a curve, the precision–recall curve in the figure. No single threshold is 'the' answer; the whole curve is the honest picture.

The headline metric: Average Precision and mAP

A whole precision–recall curve is honest but unwieldy — you cannot rank two models by eyeballing two curves. So we collapse the curve into one number: the Average Precision (AP) for a class is the area under its precision–recall curve. Because the curve already encodes the precision/recall tradeoff at every confidence threshold, its area summarises a detector's behaviour across all operating points in a single value between 0 and 1. A model that stays high-precision even as recall climbs toward 1 hugs the top-right and scores near 1; a model that must dump precision to gain any recall sags toward the origin and scores low.

Average Precision: the area under the precision-recall curve for one class.

Unpack the integral gently. As we swept the confidence threshold, recall r moved from 0 up to 1; at each recall level the curve gives a precision value, which we write as the function p(r) — 'the precision the detector achieves when its recall is r.' The integral \int_0^1 p(r)\,dr adds up p(r) across every recall from 0 to 1; geometrically that sum is the area under the curve. Why does one area summarise everything? Because it rewards a detector for holding precision high over the widest possible range of recall — exactly the all-rounder behaviour we want. If you have not met an integral before, read it as 'the average height of the precision curve as recall runs over its full range': a flat curve sitting at precision 0.8 has AP =0.8; a curve that is perfect (precision 1 everywhere) has AP =1.

One practical wrinkle you will see in papers: real precision–recall curves are jagged — precision wobbles up and down as you add predictions one by one. To make AP stable and comparable, evaluators use interpolated AP: at each recall level they replace the precision with the highest precision seen at that recall or beyond, smoothing the curve into a staircase that only steps downward. You do not need to compute this by hand; just recognise the phrase 'interpolated AP' as 'the area under the tidied-up, monotonic version of the curve' so the metric does not reward lucky local wiggles.

Mean Average Precision over C classes, with the COCO multi-threshold convention.

AP scores one class; a detector handles many, so we average. \mathrm{AP}_c is the Average Precision for class c; C is the total number of classes (80 for the COCO benchmark); the sum \sum_{c=1}^{C}\mathrm{AP}_c adds the per-class scores and the \tfrac{1}{C} turns the total into a mean — hence mean Average Precision (mAP), the single headline number every detection paper reports. Averaging over classes stops a model from looking good just by acing 'person' while flunking 'toaster'; it must perform across the board. The COCO benchmark adds a second, deeper average: instead of fixing the IoU threshold at 0.5, it computes mAP at ten thresholds — 0.50, 0.55, 0.60, \dots, 0.95 (the notation 0.50\!:\!0.05\!:\!0.95 means 'from 0.50 to 0.95 in steps of 0.05') — and averages those too.

That second average is what makes mAP so revealing, because it measures classification quality AND localization quality at once. To earn AP at IoU 0.5, a box only has to roughly land on the object — that mostly tests whether the detector named the right class and got into the neighbourhood. To still earn AP at IoU 0.9, the box must wrap the object tightly, with edges nearly on the truth — that tests precise localization. By averaging across the whole 0.50–0.95 ladder, COCO mAP rewards a detector that is both right about what (classification) and right about where, to the pixel (localization). A model can ace loose boxes yet collapse at tight ones; the multi-threshold average exposes exactly that.