Modern Detection: Going Anchor-Free and the Transformer Turn

Why retire the anchor box?

Across the last two guides one object kept doing the heavy lifting: the anchor box. Recall the idea. Instead of asking the network to invent a box out of thin air, we paper the image with thousands of pre-set reference rectangles — fixed sizes, fixed aspect ratios — at every location, and the network only has to (a) decide which anchors contain an object and (b) nudge each chosen anchor a little so it hugs the object snugly. It was a brilliant crutch. Faster R-CNN's region proposal network, and one-stage detectors like SSD and RetinaNet, all stand on it. But a crutch is still a crutch, and this final guide is about learning to walk without one.

The case against anchors has three concrete costs. First, hyperparameters: you must choose the scales, the aspect ratios, how many anchors sit at each location, and the IoU thresholds that decide which anchors count as positive (object) versus negative (background). Each one needs re-tuning per dataset — pedestrians want tall, thin anchors; overhead satellite imagery wants something else entirely — and a bad choice quietly wrecks recall. Second, density and imbalance: to cover all shapes you must tile anchors densely, tens of thousands per image, and the overwhelming majority land on background. That is exactly the foreground/background imbalance guide 4 had to fight with focal loss. Third, shape coupling: anchors bake in a prior about what shapes objects take, so an object whose aspect ratio no anchor anticipates is hard to match in the first place.

So here is the map for this guide. We first explore anchor-free detection, which keeps the familiar dense-prediction shape of a one-stage detector but predicts each box from a point rather than from an anchor. We will then notice a stubborn leftover — these models still need NMS — and that leftover becomes the springboard for the second escape route: DETR, which reframes detection as predicting a set directly, with neither anchors nor NMS. To understand DETR we will take a short, self-contained detour through attention, then assemble the whole picture.

Anchor-free detection: points and centers

If the anchor is the problem, the cleanest fix is to predict boxes straight from image locations. Two families of anchor-free detection do exactly this. The keypoint-based family finds a distinctive point — a corner or a center — and grows a box from it. The dense per-location family turns every location inside an object into a little reporter that describes the box surrounding it. Let us take them in turn; the intuition for both is wonderfully concrete.

Keypoint-based detectors like CornerNet and CenterNet lean on a heatmap — a per-pixel probability map answering 'is there a corner (or center) right here?'. CornerNet predicts two heatmaps, one for top-left corners and one for bottom-right corners, then pairs corners that belong to the same object; the box is simply the rectangle spanning a matched pair. CenterNet is even simpler: detect the object's center as a peak in a heatmap, then, at that peak, regress the width and height (and a small offset to undo rounding). The mental model is 'find a distinctive point, then grow the box around it' — no anchors, no shape priors, just peaks in a learned map.

The dense per-location family, exemplified by FCOS (Fully Convolutional One-Stage detector), is the one to study closely because it keeps the dense-grid shape you already know from YOLO and RetinaNet. The rule is disarmingly simple: any feature-map location that falls inside a ground-truth bounding box is treated as a positive sample, and that location must directly regress four numbers — the distances from itself to the box's left, top, right and bottom edges. No anchor is ever involved; the point itself is the anchor, and the four distances are the box.

FCOS box encoding: from a point's four edge-distances back to box corners.

Read this as a tiny recipe. The interior point sits at image coordinates (x, y). The network predicts four positive distances: l to the left edge, t to the top edge, r to the right edge, b to the bottom edge. To rebuild the box you walk those distances outward from the point — the left edge is at x-l, the top at y-t, the right at x+r, the bottom at y+b — giving the box corner-tuple (x-l,\,y-t,\,x+r,\,y+b). Worked example: a point at (x,y)=(50,40) predicts (l,t,r,b)=(20,20,40,40). Then the box is (50-20,\,40-20,\,50+40,\,40+40)=(30,20,90,80), i.e. left=30, top=20, right=90, bottom=80. The point describes the box from the inside out; that single change is what lets us throw the anchors away.

Centerness down-weights predictions made from points near the box boundary.

Centerness is FCOS's clever quality flag, a number between 0 and 1 that says 'how central is this point inside its box?'. Look at the ratios: \min(l,r)/\max(l,r) is 1 only when l=r (the point is exactly mid-way left-to-right) and shrinks toward 0 as the point drifts to a side edge; \min(t,b)/\max(t,b) does the same top-to-bottom. Multiplying the two and taking the square root blends both axes into one score. Worked numbers: at the exact center l=r and t=b, so centerness =\sqrt{1\cdot 1}=1. For our earlier point with (l,t,r,b)=(20,20,40,40), it is \sqrt{(20/40)\cdot(20/40)}=\sqrt{0.5\cdot0.5}=0.5. For a point hugging the left edge, say (l,t,r,b)=(5,30,55,30), it is \sqrt{(5/55)\cdot(30/30)}\approx 0.30. At training and test time this score is multiplied into the classification confidence, so wobbly, off-center boxes are pushed down the rankings and the clean, centered prediction wins.

import math

def decode_fcos(x, y, l, t, r, b):
    # Rebuild a box from a point and its four edge-distances
    left   = x - l
    top    = y - t
    right  = x + r
    bottom = y + b
    return (left, top, right, bottom)

def centerness(l, t, r, b):
    # ~1 at the box center, ~0 near a box edge
    horiz = min(l, r) / max(l, r)
    vert  = min(t, b) / max(t, b)
    return math.sqrt(horiz * vert)

# Center point of the example box (30,20,90,80) is (60,50)
print(decode_fcos(60, 50, 30, 30, 30, 30))  # (30, 20, 90, 80)
print(centerness(30, 30, 30, 30))           # 1.0  -> perfectly central
print(round(centerness(20, 20, 40, 40), 3)) # 0.5  -> off-center point

One last piece makes FCOS practical: it pairs naturally with the feature pyramid network from guide 4. A single feature map cannot cleanly handle both tiny and huge objects, and a point that falls inside two overlapping boxes is genuinely ambiguous. FCOS resolves both by letting each pyramid level handle only a band of object sizes — small objects are regressed on the high-resolution levels, large objects on the coarse levels — by capping the range of edge-distances each level is allowed to predict. Different sizes live on different floors of the pyramid, which both spreads the workload and dissolves most of the overlap ambiguity.

Still need NMS? Duplicates in anchor-free models

Here is the observation that drives the rest of the guide. Anchor-free detection removed the anchors — but it did not remove the duplicates. Think about what FCOS actually does: every location inside a ground-truth box is a positive that predicts its own box. A medium object covers dozens of feature-map locations, so dozens of neighbouring points each emit a box for the same object, all heavily overlapping. The anchors are gone, but the flood of near-identical boxes is not.

So anchor-free detectors still lean on the same cleanup we met in guide 2: non-maximum suppression. After prediction, NMS sorts boxes by score, keeps the top one, deletes every other box that overlaps it past an IoU threshold, and repeats. It works, but notice what it is — a hand-crafted, greedy, non-differentiable post-processing step bolted on outside the network, with its own threshold to tune. The network is never actually trained to output one box per object; it is trained to output many and then we tidy up afterwards. That is the last stubborn piece of the old pipeline. The next idea, DETR, finally removes it — by teaching the model to emit a clean set from the start. Hold that thought.

A crash course in attention

The detection transformer is built on transformers, which you may not have met yet on this ladder, so let us teach the one idea everything rests on — self-attention — from scratch, intuitively but precisely. The setting: you have a collection of elements (image patches, words, or — for DETR — object slots), each represented by a vector. Self-attention is the mechanism that lets each element look at all the others and update itself based on what is relevant. Nothing here requires anchors or convolutions; it is pure 'who should talk to whom'.

The trick is to give every element three learned roles. Its Query asks 'what am I looking for?'. Its Key advertises 'this is what I have to offer'. Its Value carries 'this is my actual content'. An element attends to the others by comparing its own query against everyone's keys, turning each comparison into a relevance weight, and then taking a weighted blend of everyone's values. The cleanest analogy is a soft, learned lookup table: in an ordinary dictionary you match a key exactly and grab its value; in attention you match a query against all keys by degree of similarity and return a smooth mixture of all the values, weighted by how well each key matched.

Scaled dot-product attention — the whole mechanism in one line.

Let us dissect it. Q, K, V are matrices: stack every element's query vector as a row of Q, every key as a row of K, every value as a row of V, with each vector of length d_k. The product QK^{\top} multiplies every query by every key, producing a square table of all pairwise similarity scores — entry (i,j) is how strongly element i's query matches element j's key (a large dot product means 'very relevant'). Dividing by \sqrt{d_k} is a stabiliser: dot products of long vectors grow with dimension, and without this the scores would get so large that softmax saturates into a near one-hot spike and gradients vanish; if d_k=64 we divide by 8. The \operatorname{softmax} runs along each row, squashing that row of scores into positive weights that sum to 1 — a proper attention distribution. Finally multiplying by V replaces each element with the weighted blend of all values, weighted by those attention weights. Tiny example: if one query scores (2.0,\,0.1,\,0.3) against three keys, softmax gives roughly (0.75,\,0.11,\,0.14), so the output is about 0.75\,V_1+0.11\,V_2+0.14\,V_3 — mostly the first element's content, with a touch of the others.

DETR: detection as set prediction

Now we assemble everything. The detection transformer (DETR) reframes object detection as direct set prediction, and its pipeline has four stages. First, a CNN backbone turns the image into a grid of feature vectors, just as in every detector so far. Second, a transformer encoder applies self-attention over those features so each location is enriched by global context — the model can relate a wheel to a car body across the whole image. Third, a transformer decoder takes a small fixed number N of learned vectors called object queries (say N=100) and, using attention, lets each query gather evidence from the encoded image and from the other queries. Fourth, every query is passed through a small head that outputs exactly one prediction: a class label (one of the real classes, or a special 'no object' label) and a box. So DETR emits a set of N predictions in one shot — no anchors, no sliding windows, and, as we will see, no NMS.

What are object queries, really? Think of them as N learned slots, empty containers the network trains from scratch. Through attention each slot learns to specialise — one slot tends to look for objects in the upper-left, another for small objects, another for wide ones — so collectively they spread out to cover the image's regions and shapes. A slot is not a box and not an anchor; it is a learned 'question about the scene' that, after gathering evidence, resolves into at most one detected object. The slots also attend to each other in the decoder, which lets them coordinate: 'you take that car, I'll take the person'.

Set prediction via optimal bipartite (Hungarian) matching.

This loss is the heart of DETR, so let us read it slowly. The model outputs N predictions \hat{y}=(\hat{y}_1,\dots,\hat{y}_N). The ground-truth objects are y=(y_1,\dots,y_N), padded out to length N with copies of the 'no object' label so the two lists are the same size. A permutation \sigma is just a one-to-one pairing — it says 'ground truth i is handled by prediction number \sigma(i)' — and \mathfrak{S}_N is the set of all such pairings. For each candidate pairing we sum a matching cost \mathcal{L}_{\text{match}}(y_i,\hat{y}_{\sigma(i)}) that blends a class term (is the predicted class right?) with a box term (typically L1 distance plus generalized IoU — how well do the boxes overlap?). We then pick \hat{\sigma}, the pairing with the smallest total cost — that is the \arg\min. Crucially the assignment is one-to-one: each ground-truth object is claimed by exactly one prediction, and every leftover prediction is matched to 'no object'.

Two things make this work and matter. First, searching all N! permutations by brute force is hopeless, but this is a classic assignment problem, and the Hungarian algorithm solves it exactly in O(N^3) time — fast and standard. Second, and this is the punchline of the whole guide: because the matching is one-to-one, the training signal tells each ground-truth object 'exactly one of you is responsible for me'. A query that produces a duplicate of an object already claimed by another query gets no reward — its best move is to predict 'no object' instead. The model is therefore trained to not emit duplicates. That is precisely why DETR needs no non-maximum suppression: the redundancy NMS used to delete after the fact is never produced in the first place. The non-differentiable post-processing step from guide 2 dissolves into the differentiable loss.

# DETR training loss, in pseudocode
#   preds: N predictions, each (class_logits, box)
#   gts:   M ground-truth objects, each (class, box), M <= N

def detr_loss(preds, gts):
    # 1) Build an N x N cost matrix (gts padded with 'no object')
    cost = build_cost_matrix(
        preds, gts,
        class_weight=1.0,      # is the class right?
        l1_weight=5.0,         # box corner distance
        giou_weight=2.0,       # box overlap (generalized IoU)
    )
    # 2) Optimal one-to-one assignment (no NMS, no anchors)
    pred_idx, gt_idx = hungarian_algorithm(cost)   # O(N^3)
    # 3) Supervise each matched pair; unmatched preds -> 'no object'
    loss = 0.0
    for p, g in zip(pred_idx, gt_idx):
        loss += classification_loss(preds[p], gts[g])
        if gts[g].label != NO_OBJECT:
            loss += l1_box_loss(preds[p], gts[g])
            loss += giou_loss(preds[p], gts[g])
    return loss

The state of detection: a unified view

Step back and the whole track tells one story. Object detection is localization plus classification — saying both what and where (guide 1). To grade and compare detectors we built a shared scoring toolkit: IoU to measure box overlap, NMS to remove duplicates, and mAP to summarise quality across thresholds (guide 2). Then we walked the architectures. The two-stage Faster R-CNN family proposes then classifies: first sketch promising regions, then label and refine them — accurate but heavier (guide 3). The one-stage YOLO/SSD/RetinaNet family predicts densely in a single pass, leaning on anchors, the feature pyramid, and focal loss to stay both fast and accurate (guide 4). And in this guide the field shed its hand-designed scaffolding: anchor-free detectors predict from points, and DETR predicts a set directly with Hungarian matching (guide 5).

1. Define the task: predict a class label and a box for every object — localization plus classification.
2. Score it: IoU for overlap, NMS to deduplicate, mAP to rank detectors fairly.
3. Two-stage: propose regions, then classify and refine (R-CNN to Faster R-CNN).
4. One-stage: dense prediction with anchors, FPN, and focal loss (YOLO, SSD, RetinaNet).
5. Modern: drop anchors (point-based, FCOS) and drop NMS too (DETR set prediction).

Two tensions run through the whole arc, and neither is 'solved' — you simply choose your point on them. The first is speed versus accuracy: two-stage detectors and large transformers buy accuracy with compute; one-stage and real-time models buy latency with some accuracy; the right pick depends on whether you are labelling a research benchmark or running on a phone or a car. The second is hand-crafted versus learned components: each generation has replaced a human-designed piece — sliding windows, then proposals, then anchors, then NMS — with something the network learns end-to-end. The trajectory is clear: fewer hand-tuned knobs, more learned behaviour, simpler pipelines that are harder to break by mis-tuning.

Where is the field heading? Three currents are worth watching. Real-time transformer detectors (RT-DETR and kin) are closing the speed gap, so set prediction is no longer a research-only luxury. Open-vocabulary and promptable detection — models like Grounding DINO, OWL-ViT, and the Segment Anything family — let you ask for objects by free-text name or by a clicked point, rather than from a fixed list of training classes; detection is becoming something you talk to. And detection is unifying with segmentation and tracking: the same query-and-set machinery that finds boxes also produces masks and follows objects through video, so the boundaries between these once-separate tasks are dissolving into one general 'find the things' interface.