Dropping the proposal stage
In the previous guide we built up the two-stage family — R-CNN through Faster R-CNN. Those detectors are accurate, but they work in two passes: first a region proposal network suggests a few hundred 'maybe there's an object here' boxes, then a second network crops and classifies each one. That propose-then-classify split is the bottleneck. Even Faster R-CNN, the fastest of the family, runs at only a handful of frames per second — fine for labelling a photo album, far too slow for live video, a robot reacting to its surroundings, or a phone app that must run on a tiny battery-powered chip.
The one-stage idea is radical in its simplicity: throw away the proposal step. Instead of asking 'where might objects be?' and then 'what are they?', a one-stage object detection model answers both questions at once, directly, for every location on a dense grid laid over the feature map. Picture the convolutional feature map you met in earlier tracks — a grid of cells, each summarising a patch of the image. A one-stage detector attaches a small prediction head to that grid so that at every cell it emits, in a single forward pass, both a class score and a box. Some designs sharpen this further by giving each cell a few anchor boxes (preset reference shapes) to refine, an idea we deepen in Section 3.
Diagram of a feature-map grid with a prediction head producing class scores and box coordinates at each grid cell.
YOLO: detection as grid regression
The first detector to make this work end-to-end was YOLO — 'You Only Look Once'. Its recipe is beautifully direct. Divide the input image into an S×S grid. The cell that contains the centre of an object becomes responsible for detecting that object. Each cell predicts B candidate bounding boxes, each described by five numbers — x, y, w, h, and a confidence — plus one shared set of C class probabilities for the whole cell. So the cell's job is: 'If there's an object centred in me, here are a couple of guesses at its box, how sure I am, and what I think it is.'
The full YOLO output is one tensor of this shape.
Read that shape carefully — it is the whole model output in one line. S×S is the grid (the original YOLO used S=7, so 49 cells). For each cell we store B·5 + C numbers. B is how many boxes the cell predicts (YOLO used B=2); the 5 is the per-box bundle (x, y, w, h, confidence); and C is the number of object classes (C=20 on PASCAL VOC). Plugging in: 7×7×(2·5 + 20) = 7×7×30. That single 7×7×30 tensor is the entire prediction for the image — no proposals, no second pass. Note the asymmetry baked in here: each cell predicts B boxes but only ONE shared set of class probabilities, so a single cell can ultimately commit to only one class. This is exactly why early YOLO struggled when two objects of different classes shared a cell.
What does that per-box 'confidence' mean, exactly? YOLO defines it as Pr(object) × IoU — the probability that a box contains any object at all, multiplied by how well the box is expected to overlap the true object (its intersection over union). So confidence rolls 'is something here?' and 'is my box tight?' into one number; an empty cell should drive it to zero. To turn this into a usable, class-specific score at test time, you multiply: class-specific score = confidence × class probability. A box with confidence 0.85 and a 'dog' probability of 0.90 yields a final dog score of 0.85 × 0.90 = 0.765 — and that is the number NMS and mAP from guide 2 will operate on.
# Decoding one YOLO grid cell (PASCAL VOC config: S=7, B=2, C=20) # Image is 448x448, so each cell covers 448/7 = 64 px. # Look at cell (row=3, col=4) and its best of the B boxes: tx, ty, tw, th, conf = 0.30, 0.60, 0.50, 0.40, 0.85 # raw box predictions p_dog = 0.90 # top class probability for this cell # x, y are offsets WITHIN the cell (0..1); w, h are fractions of the whole image. cell = 448 / 7 # 64 px per cell cx = (4 + tx) * cell # center x = (4 + 0.30) * 64 = 275.2 px cy = (3 + ty) * cell # center y = (3 + 0.60) * 64 = 230.4 px bw = tw * 448 # width = 0.50 * 448 = 224.0 px bh = th * 448 # height = 0.40 * 448 = 179.2 px # Final class-specific score = box confidence * class probability score_dog = conf * p_dog # 0.85 * 0.90 = 0.765 -> fed to NMS / mAP
YOLO's training loss: localisation + objectness + no-object + classification.
It looks busy, but it is just a sum of squared-error terms, each guarded by an indicator. 𝟙_ij^obj is 1 when box j in cell i is responsible for a real object (0 otherwise); 𝟙_ij^noobj is its opposite. Hatted symbols are predictions, un-hatted are targets. Line 1 penalises the box centre (x, y). Line 2 penalises width and height — but through their square roots √w, √h, a deliberate trick: a 10-pixel error on a tiny box matters far more than on a huge one, and the square root compresses large sizes so small boxes get fair weight. Line 3 pushes the confidence C of object boxes toward their true IoU; line 4 pushes empty-box confidence toward 0; line 5 is classification error, applied only in cells that contain an object. The two knobs set priorities: λ_coord = 5 says 'getting boxes right matters a lot', while λ_noobj = 0.5 down-weights the no-object term. Why down-weight it? On a 7×7 grid most cells are empty, so the no-object term appears far more often than anything else; left unchecked the network would minimise the loss by predicting 'nothing here' everywhere and swamp the gradient from the rare cells that hold objects. λ_noobj is a blunt fix for that imbalance — hold the thought, because Section 5 cures the very same disease far more elegantly.
Several differently-shaped dashed reference boxes (anchors) tiled at one grid cell, each adjustable toward an object.
Anchors and SSD: multiple shapes, multiple scales
YOLO's cells predict boxes from scratch, which is hard — the network must learn the full range of object shapes with no head start. Anchor boxes, which you first met in guide 3, give the dense grid a head start: a small, fixed vocabulary of reference boxes — several aspect ratios (tall, square, wide) at several scales — is tiled at every location of the feature map. The network no longer invents a box from nothing; for each anchor it predicts just an offset that deforms the anchor toward a nearby real object, plus a class score for what sits inside. Refining a good starting guess is far easier than regressing absolute coordinates.
A grid cell overlaid with multiple dashed rectangles of varying aspect ratio and size representing anchors.
How does the network learn which anchor should chase which object? Through training-time anchor matching by intersection over union. For each ground-truth box, we compute its IoU with every anchor and assign the object to the anchor (or anchors) that overlap it most — those become positive anchors tasked with predicting this object's class and offset. Anchors with low IoU against every object are labelled negative (background), and anchors in the ambiguous middle are usually ignored. So each anchor specialises: a tall anchor learns to catch people and lamp-posts, a wide anchor learns to catch cars and buses.
Turning predicted offsets (t_x, t_y, t_w, t_h) into a real box (b).
This is how the four raw offsets become a real box, and it is the exact inverse of the regression targets from guide 3. Start with an anchor at grid location (c_x, c_y) with preset size (p_w, p_h). The network outputs four numbers (t_x, t_y, t_w, t_h). For the centre we pass t_x, t_y through the sigmoid σ, which squashes any real number into the range 0–1; adding that to the cell coordinate c_x, c_y keeps the predicted centre inside its own cell, so an anchor can't run off to claim a far-away object — a crucial stabiliser during training. For the size we multiply the anchor's width and height by e^{t_w} and e^{t_h}. The exponential is always positive, so a box can never get a negative width, and it makes scaling multiplicative: t_w = 0 leaves the anchor unchanged (e^0 = 1), t_w = +0.69 doubles it (e^0.69 ≈ 2), t_w = −0.69 halves it. Training simply inverts these formulas — given a matched anchor and its ground-truth box, it computes the target t's the network should aim for.
SSD — the Single Shot MultiBox Detector — adds the second big idea: don't predict from just one feature map, predict from several at once. A deep convolutional backbone naturally produces a stack of feature maps that get smaller and coarser as you go deeper. SSD attaches anchor-and-class prediction heads to several of these layers. The shallow, high-resolution maps still see fine detail, so their anchors catch small objects; the deep, low-resolution maps each summarise a large region, so their anchors catch big objects — all in the same single forward pass. This multi-scale prediction is exactly the weakness one-stage detectors needed to fix, and it foreshadows the feature pyramid in the next section.
Seeing at every scale: Feature Pyramid Networks
SSD's multi-layer trick exposes a deeper dilemma about scale. As an image passes through a deep backbone, each successive feature map is more semantically rich — deeper layers have combined edges into textures into parts into whole-object concepts — but also spatially coarser, because pooling and striding shrink the map. So deep maps know 'this is a dog' but only at a blurry resolution: great for big objects, nearly blind to small ones. Shallow maps are the opposite: spatially fine, with crisp locations, but semantically weak — they see edges and blobs, not 'dog'. SSD's small-object predictions come from exactly those shallow, semantically weak layers, which limits how well it does on them.
The Feature Pyramid Network (FPN) resolves the dilemma instead of choosing a side. On top of the usual bottom-up backbone it adds a top-down pathway: take the deepest, most semantic map, upsample it back to a larger size, and merge it with the shallower same-resolution map from the backbone through a lateral connection (a 1×1 convolution that lines up the channels, then an element-wise add). Repeat down the stack. The result is a pyramid of feature maps that are BOTH spatially fine AND semantically strong at every level — the shallow maps inherit the deep maps' 'dog-ness' while keeping their own sharp locations.
A stack of progressively smaller feature maps produced by a convolutional backbone.
An analogy: think of a stack of maps of the same city — a country-level map, a region map, a city map, a street map. The street map is detailed but unlabelled; the country map is richly labelled but too zoomed-out to place anything precisely. FPN is like going back and copying the country map's labels onto every finer map, so the street map now carries both precise streets and rich names. Concretely, this connects to receptive fields from earlier tracks: a unit deep in the network has a large receptive field (it 'sees' a big chunk of the image, good for big objects), while a shallow unit has a small receptive field (good for small objects). FPN lets a detector place anchor boxes on each pyramid level matched to that level's scale — small anchors on fine levels, large anchors on coarse levels.
Diagram showing a small input patch expanding to a large receptive field through successive convolutional layers.
The class-imbalance killer: Focal Loss and RetinaNet
Even with FPN, one-stage detectors still trailed two-stage accuracy, and the authors of RetinaNet found the true culprit was the loss, not the architecture. A dense one-stage detector places a huge number of anchor boxes — tens of thousands per image, often 10^4 to 10^5. In any given image only a few dozen of them actually overlap a real object; everything else is background. Two-stage detectors never face this flood, because the proposal stage already filtered the candidates down to a few hundred roughly balanced ones. The one-stage detector sees the raw, wildly imbalanced sea.
Here's why that breaks ordinary cross-entropy. Each background anchor is easy — the network quickly learns to say 'background' with, say, 90% confidence — so each one's individual loss is small. But there are tens of thousands of them, and a great many small losses still sum to a mountain that buries the loss from the few hard, interesting examples (a half-hidden object, a rare class). The gradient ends up pointing wherever the easy negatives want it to go; the rare positives barely move it. The network optimises the wrong thing and accuracy stalls. YOLO's λ_noobj was a crude attempt at this; we can do far better.
Focal loss = cross-entropy x a modulating factor that mutes easy examples.
Focal loss is just cross-entropy with one extra factor bolted on. Read the pieces: p_t is the model's predicted probability for the TRUE class — high when the model is right and confident, low when it's wrong. The term −log(p_t) alone is ordinary cross-entropy. The new factor (1 − p_t)^γ is the modulating factor, and γ ≥ 0 is the focusing parameter that controls how aggressively easy examples are muted. α_t is a simple class-balancing weight (a typical value is 0.25) that further tempers the positive/negative ratio. The magic is in (1 − p_t)^γ: for a well-classified example p_t is near 1, so (1 − p_t) is near 0, and raising that to the power γ drives the factor toward 0 — the easy example's loss is suppressed. For a hard, misclassified example p_t is small, (1 − p_t) is near 1, and the factor stays near 1, so its loss is left almost untouched.
Bolt the pieces together and you get RetinaNet = a backbone (e.g. ResNet) + an FPN neck for multi-scale features + dense anchors on every pyramid level + focal loss to tame the imbalance. The headline result silenced the long-standing assumption that one-stage meant 'fast but less accurate': RetinaNet matched or beat the best two-stage detectors of its day on accuracy while running faster. The lesson is striking — the bottleneck wasn't the missing proposal stage at all; it was a loss that let easy background drown out the signal. Fix the loss, and the speed of one-stage came for free.
One-stage vs two-stage: choosing a detector
Let's consolidate the whole detection family along four axes that matter in practice: speed, accuracy, how well it handles small or densely-packed objects, and implementation complexity. Keep in mind these all shift over time as architectures improve — what follows is the classic, era-defining comparison that explains the design space, not a fixed leaderboard.
Faster R-CNN (two-stage): the accuracy benchmark, and historically the strongest on small and crowded objects because its proposal stage and per-region refinement give each candidate careful attention. The cost is speed — two passes and per-region computation — and the most moving parts. Pick it when accuracy is paramount and you have a server-class GPU and some latency budget. SSD (one-stage, multi-scale anchors): much faster than Faster R-CNN and reasonably accurate, predicting from several feature maps in one pass. Without FPN its small-object accuracy is middling, and it is sensitive to how its anchors are configured. A solid middle-ground choice.
YOLO (one-stage, grid): the speed champion and the natural pick for real-time use. Later versions (v3 onward) added anchors and multi-scale prediction and closed much of the early small-object gap, but its design philosophy stays 'fast first.' SSD and YOLO are the two you'll reach for on edge devices. RetinaNet (one-stage, FPN + focal loss): the accuracy-minded one-stage detector — it proved you can have two-stage-level accuracy at one-stage speed, at the cost of being a bit slower than YOLO. Reach for it when you want strong accuracy but can't afford a two-stage pipeline.
But notice the thread running through every fast detector in this guide: anchor boxes. Anchors are powerful, yet they drag in a pile of hand-tuned hyperparameters — how many scales, which aspect ratios, how many anchors per location, and the IoU thresholds that decide matching — and the best settings differ from dataset to dataset, so they must be re-tuned every time. Worse, after all that, every detector here still leans on NMS (from guide 2) as a separate, heuristic post-processing step to clean up duplicate boxes. It's natural to ask: could we detect objects without anchors at all — and even without NMS? That question drives the final guide, where we go anchor-free and meet the transformer-based detectors that reframe detection as direct set prediction.