Two-Stage Detectors: From R-CNN to Faster R-CNN

The big idea: propose, then classify

Imagine you wanted to find every object in a photo with no clever tricks at all. The obvious recipe is a sliding window: slide a little box across every position in the image, and at each stop ask 'is there an object here?' But objects come in many sizes and shapes, so you would have to repeat this for boxes of every scale (small, medium, huge) and every aspect ratio (tall, square, wide). Multiply the number of positions by the number of scales by the number of shapes and you get hundreds of thousands to millions of boxes per image — and you would run a classifier on each one. That is astronomically expensive, and almost all of those boxes are empty sky or blank wall. Object detection needs a smarter plan.

The two-stage idea is disarmingly simple: split the work into propose, then classify. First, a fast process scans the image and returns a few hundred to a few thousand candidate boxes — places where something interesting might be, regardless of what it is. Each such box is a region proposal: a class-agnostic 'something might be here' rectangle. It carries no opinion about whether the thing is a dog, a car, or a teapot; it only flags a region worth a closer look. Second, we spend real computational effort only on those few promising regions — classifying each one and refining its box. We have traded an unwinnable search over millions of windows for a careful study of a manageable shortlist.

Everything that follows is the story of making each half of that split better and cheaper. R-CNN proved the idea works but was painfully slow; Fast R-CNN shared computation so the 'classify' stage flew; Faster R-CNN learned the 'propose' stage instead of hand-designing it; and RoI Align fixed a small but costly alignment bug. By the end you will be able to trace a single image all the way from raw pixels to a final list of detected objects, and understand why each piece is there.

R-CNN: regions plus a CNN classifier

In 2014, R-CNN (Regions with CNN features) was the first method to bring the raw power of convolutional networks to detection, and it did so by bolting a CNN onto the propose-then-classify skeleton. The 'propose' step used a classic, hand-designed algorithm called Selective Search, which groups together neighbouring pixels with similar colour and texture to suggest roughly 2000 region proposals per image. No learning is involved in that step — it is pure image processing — but it reliably casts a net wide enough to cover most real objects.

1. Propose: run Selective Search on the image to get ~2000 class-agnostic candidate boxes.
2. Warp: crop each proposal out of the image and stretch it to a fixed input size (e.g. 224x224), regardless of its original shape.
3. Extract: pass each warped crop through the CNN to get a fixed-length feature vector.
4. Classify: feed that vector to a set of class-specific linear SVMs, one per object category, to score 'is this a dog? a car? ...'.
5. Refine: a separate linear regressor nudges the proposal's box to fit the object more tightly.

One R-CNN idea deserves special attention because it recurs everywhere later: bounding-box regression. Rather than ask the network to predict a box's absolute pixel coordinates from scratch, we ask it to predict small corrections (offsets) relative to the proposal it already has. A proposal is usually close to the true object, so the correction is small — and predicting a small nudge is far easier and more stable than conjuring raw coordinates out of thin air. Concretely, given a proposal with centre (x_a, y_a) and size (w_a, h_a), and a ground-truth box with centre (x, y) and size (w, h), the regression targets are defined as follows. Recall a bounding box is just four numbers describing a rectangle.

Box-regression targets: centre offsets normalized by proposal size, and log-ratios for width and height.

Read this term by term. The symbols x_a, y_a, w_a, h_a are the anchor/proposal centre coordinates and its width and height; x, y, w, h are the true object's centre, width, and height; and t_x, t_y, t_w, t_h are the four numbers the network is trained to output. For the centre, t_x = (x - x_a)/w_a measures how far the true centre sits from the proposal's centre, expressed in units of the proposal's own width. Dividing by w_a is what makes the target scale-invariant: a 5-pixel shift on a tiny 10-pixel box is a big deal (t_x = 0.5), while the same 5 pixels on a 500-pixel box barely matters (t_x = 0.01). For size, t_w = log(w/w_a) uses a logarithm so that growing and shrinking are symmetric: doubling the width gives log 2 ≈ +0.69, halving it gives log(1/2) ≈ -0.69 — equal and opposite. (A plain ratio would squeeze shrinking into the cramped interval 0–1 but let growing run to infinity, an ugly imbalance for a network to learn.) The log also guarantees the predicted width stays positive once we invert it with an exponential. Worked example: a proposal centred at (100, 100) of size 50x50, with a true box centred at (110, 95) of size 60x40, gives t_x = 10/50 = 0.2, t_y = -5/50 = -0.1, t_w = log(1.2) ≈ 0.18, t_h = log(0.8) ≈ -0.22 — four small, well-behaved numbers, exactly the kind a network learns happily.

Fast R-CNN: share the convolutions

R-CNN's fatal waste is obvious once you see it: ~2000 crops from one image overlap heavily, so the CNN re-computes nearly identical features thousands of times. Fast R-CNN (2015) fixes this with one decisive insight — run the backbone CNN once over the whole image to produce a single shared feature map, and then, for each proposal, crop the corresponding region out of that feature map instead of re-running the CNN on a pixel crop. Because convolution preserves spatial layout, a box in the original image corresponds to a matching box in the feature map; we just look up the right rectangle of already-computed features. The expensive convolutional work is now done a single time, no matter how many region proposals there are.

But there is a snag. A detection head — the small classifier-and-regressor at the end — expects a fixed-size input, yet proposals come in every imaginable shape, so the patches we crop from the feature map are all different sizes. Fast R-CNN's answer is RoI pooling ('RoI' = Region of Interest). It divides whatever-sized feature patch into a fixed grid (say 7x7 cells), then max-pools within each cell, always emitting the same 7x7 output no matter how big or small the input region was. A 35x21 patch and a 14x7 patch both come out as a tidy 7x7 block the head can swallow. (We will see in the RoI Align section that this rounding-to-a-grid step has a subtle flaw that RoI Align later repairs.)

With fixed-size features in hand, Fast R-CNN attaches a multi-task head: two sibling output layers fed by the same shared features. One outputs a probability distribution p over the K object classes plus a 'background' class; the other outputs box-regression offsets t (the t_x, t_y, t_w, t_h from the previous section) for each class. Crucially, both are trained together in one shot with a single combined loss, end to end — everything except the proposals, which still come from Selective Search. That joint loss is:

Fast R-CNN's multi-task loss: classification plus a switched-on-only-for-objects localization term.

Symbol by symbol: p is the predicted class probability distribution and u is the true class label (an integer; by convention u = 0 means background and u ≥ 1 means a real object class). L_cls(p, u) is the classification loss — typically the negative log-probability the model assigned to the correct class, so it is small when the model is confidently right. t is the predicted box-offset 4-tuple for the true class, and t* is the target offset computed from the ground-truth box exactly as in the previous section; L_loc(t, t*) is the localization loss, a smooth-L1 (Huber) distance that behaves like squared error for small mistakes but like absolute error for large ones, which keeps a single badly-placed proposal from blowing up the gradients. The bracket [u ≥ 1] is an indicator: it equals 1 for a real object and 0 for background, so background RoIs contribute no localization loss at all. That is the whole point — there is no 'correct box' to regress toward when the region is just empty wall, so we must not ask the regressor to fit one. Finally, λ is a single positive number that balances the two tasks; with λ = 1 (a common choice) classification and localization carry equal weight. Set λ too high and the model chases pixel-perfect boxes while mislabelling them; too low and it labels well but localizes loosely. Example: for a background RoI, [u ≥ 1] = 0, so L = L_cls alone; for a 'cat' RoI with u = 3, both terms are active and the box is pushed to fit.

Faster R-CNN: learn the proposals too

Fast R-CNN made the 'classify' stage lightning-quick, which exposed an awkward truth: the 'propose' stage was now the slow part. Selective Search runs on the CPU, takes roughly two seconds per image, and — worse conceptually — it is hand-designed and frozen, unable to learn from data or improve with the rest of the system. Faster R-CNN (2015) delivers the final blow by learning the proposals too, folding them into the same network so the whole detector becomes one trainable unit.

Its new ingredient is the Region Proposal Network (RPN): a small, fully-convolutional network that slides over the very same shared feature map the detection head uses. At each location on that feature map, the RPN looks at a tiny window of features and emits two things — an objectness score ('how likely is there any object here, versus background?') and a set of box offsets to refine the region. Because it is just a couple of convolutional layers riding on top of features the backbone already computed, generating all the region proposals is nearly free, and it runs on the GPU alongside everything else.

But a single point on a feature map has no width or height — so how can it predict a box? This is where anchor boxes enter, and they are worth meeting slowly because they power most detectors that follow. The idea: at every feature-map location we pre-place a fixed set of reference rectangles — the anchor boxes — of several scales (small, medium, large) and several aspect ratios (e.g. 1:1, 1:2, 2:1). A typical setup uses 3 scales x 3 ratios = 9 anchors per location. Each anchor is a fixed, known box, so the RPN's job becomes beautifully concrete: for each anchor, predict (a) an objectness score saying whether a real object overlaps this particular anchor, and (b) four offsets (t_x, t_y, t_w, t_h) that deform the anchor into a tighter fit. The anchors tile the image at many sizes and shapes, so collectively they can match almost any object, while each individual prediction is just the small, easy correction we praised earlier.

How is the RPN trained? With exactly the loss form you just learned — a classification term plus a switched-on-only-for-objects regression term — just specialised to two classes. An anchor is labelled a positive (object) example if it overlaps a ground-truth box well (high IoU, the overlap measure from Guide 2) and a negative (background) example if it overlaps nothing; the objectness score is trained with that binary classification loss, and the offsets with the smooth-L1 localization loss, which (as before) is computed only for positive anchors. The RPN's top few hundred-to-thousand highest-scoring proposals are then passed downstream to the Fast R-CNN detection head — the second stage — for full classification and a final box refinement.

RoI Align: fixing the alignment problem

Now back to that 'subtle flaw' flagged earlier. Recall that RoI pooling must take a proposal — whose coordinates are real-valued floats living in the original image — and read features from the discrete integer grid of the feature map. To do this it rounds (quantizes) twice: first when snapping the proposal's borders onto the nearest feature-grid cells, and again when dividing that region into the fixed 7x7 sub-cells (whose boundaries are rounded too). Each rounding can shift the sampled region by up to half a feature-map cell, which — because the feature map is downsampled, often by 16x or 32x relative to the image — can mean a misalignment of many pixels back in the original image. Faster R-CNN's classifier barely notices, because deciding 'cat vs dog' is robust to a small spatial shift. But anything needing precise spatial fidelity — tight boxes, and especially per-pixel masks — suffers.

RoI Align (introduced with Mask R-CNN in 2017) removes all rounding. It keeps the proposal's exact floating-point borders, divides them into the fixed grid with float boundaries, and places sampling points at precise sub-pixel locations inside each cell. The catch: a sampling point like (2.7, 4.2) lands between the integer feature cells, where there is no stored value. RoI Align computes the value there by bilinear interpolation — a weighted blend of the four nearest grid values. For a point (x, y) sitting between integer cells, let a and b be its fractional offsets; then:

Bilinear interpolation: a weighted blend of the four surrounding feature values, with weights that sum to 1.

Unpacking it: ⌊x⌋ is the floor (round-down) of x, so a = x - ⌊x⌋ is how far past the left grid line our point sits, a fraction between 0 and 1; b is the same fraction in the vertical direction. The four values f_00, f_10, f_01, f_11 are the stored features at the four integer corners surrounding the point — bottom-left, bottom-right, top-left, top-right. Each corner's weight is the product of the two fractional distances to the opposite corner: the bottom-left corner gets (1-a)(1-b), large only when the point is close to it (a and b both small). So nearer corners get bigger weights — exactly the intuition that a value should look most like the data point it is closest to. The four weights always sum to 1, so the result is a true weighted average, never inflated or shrunk. Worked example at (x, y) = (2.7, 4.2): a = 0.7, b = 0.2, giving weights (1-0.7)(1-0.2)=0.24, (0.7)(0.8)=0.56, (0.3)(0.2)=0.06, (0.7)(0.2)=0.14 — they sum to 1.00, and the corner at x = 3 (the f_10 corner) dominates at 0.56 because the point sits closer to the right. Because no coordinate is ever rounded, the sampled features stay locked to the true region, preserving spatial fidelity down to the sub-pixel.

Putting it together: the two-stage pipeline

Let us trace one image through the finished Faster R-CNN (with RoI Align) from end to end, so every piece lands in its place. The pipeline is a clean assembly line in which the expensive convolutional work happens once and the cheap per-region work happens many times.

1. Backbone: feed the whole image through the CNN once to produce a single shared feature map.
2. RPN: slide the Region Proposal Network over that feature map; at each location score the anchor boxes for objectness and predict their offsets, yielding a few hundred-to-thousand proposals.
3. RoI Align: for each surviving proposal, sample a fixed-size (e.g. 7x7) feature grid out of the shared map with no rounding.
4. Detection head: classify each region into one of the K classes (or background) and regress a final, refined box.
5. NMS: remove duplicate, overlapping detections of the same object, keeping the highest-scoring one.
6. Output: the surviving boxes-with-labels are the final detections; their quality is summarised by mAP.

# Faster R-CNN forward pass (inference), pseudocode
def detect(image):
    feat = backbone(image)              # 1. one shared feature map

    # 2. RPN proposes regions from anchors
    objectness, rpn_offsets = rpn(feat)             # score + box delta per anchor
    proposals = decode(anchors, rpn_offsets)        # anchor + offset -> box
    proposals = top_k(proposals, objectness, k=1000)
    proposals = nms(proposals, iou_thresh=0.7)      # thin out RPN duplicates

    detections = []
    for box in proposals:               # cheap per-region work
        roi = roi_align(feat, box, out=(7, 7))      # 3. no rounding
        cls_probs, box_offsets = head(roi)          # 4. what + where
        label, score = argmax(cls_probs)
        if label != BACKGROUND and score > thresh:
            refined = decode(box, box_offsets[label])
            detections.append((label, score, refined))

    # 5. drop duplicate detections of the same object
    return nms_per_class(detections, iou_thresh=0.5)

What did this lineage buy us? Two-stage detectors hold a reputation for top-tier accuracy, and they are especially strong on small and densely packed objects, because the Region Proposal Network can nominate many overlapping candidates and the dedicated second stage examines each closely. The cost is structural: two stages run in sequence — propose, then classify — which adds latency and makes hitting strict real-time budgets (say, 30+ frames per second on video) genuinely hard. Note too that non-maximum suppression from Guide 2 appears twice in the pipeline above (once to thin RPN proposals, once on the final detections), a reminder that the scoring tools you already learned are load-bearing parts of any real detector.

That speed tax raises an irresistible question: must we really propose first and classify second? What if a single network could look at the image once and directly emit boxes-with-labels in one shot — trading a little accuracy for a lot of speed? That is exactly the bet the one-stage detectors make, and it is where we head next: YOLO, SSD, and the clever focal loss that lets a one-pass detector rival the two-stage giants.