detection transformer

DETR — DEtection TRansformer (Carion et al., 2020) — reframes object detection as direct set prediction: the model outputs a fixed-size set of predictions in one shot, and is trained to make that set match the ground-truth objects as a whole. This is a clean break from everything before it. There are no anchors, no region proposals, and crucially no non-maximum suppression — the entire hand-engineered pipeline of candidate generation and duplicate removal is replaced by an end-to-end learned model.

The architecture combines a CNN backbone with a transformer encoder-decoder. The backbone extracts features; the encoder uses self-attention to let every feature location attend to every other, giving global context. The decoder takes a small fixed number (e.g. 100) of learned 'object queries' — slots that each learn to look for one object — and, via cross-attention to the encoded image, transforms each query into either a (class, box) prediction or a 'no object' (∅) output. N is chosen larger than the most objects ever expected in an image.

The training engine is bipartite matching. Because the output is an unordered set, DETR first finds the single best one-to-one assignment between its N predictions and the ground-truth objects (padded with ∅), using the Hungarian algorithm to minimize a matching cost combining class probability and box (L1 + GIoU) error. Then it backpropagates a loss only against that optimal assignment. This one-to-one matching is exactly what eliminates duplicates: since each object is matched to a single prediction, the model is penalized for emitting two boxes for one object, so it learns not to — no NMS required.

The original DETR was elegant but had drawbacks: very slow convergence (500 epochs) and weak small-object accuracy, because global attention over high-resolution features is expensive and queries take long to specialize. Follow-ups fixed these: Deformable DETR attends to only a few sampled points (faster, multi-scale, better on small objects); DINO and DN-DETR add denoising and better query designs to reach state-of-the-art. DETR's lasting impact is conceptual — it showed detection can be a fully end-to-end, NMS-free, attention-based set-prediction problem, and seeded the query-based paradigm now spanning segmentation, tracking, and open-vocabulary detection.