What Is Object Detection? Finding the What and the Where

From 'what' to 'what and where'

Earlier in this ladder you met image classification: you feed in one image and the network hands back one label. Under the hood that label is a fixed-length probability vector — say 1000 numbers for 1000 categories — and you simply read off the largest one. The crucial detail is that the shape of the answer never changes. Whether the photo shows a cat, a fire truck, or a bowl of soup, the output is always exactly one vector of the same length. The question being answered is narrow: 'Of my known categories, which single thing is this picture mostly about?'

Object detection raises the bar. Instead of one label for the whole image, it must produce a list of items, where each item is a triple: a class label (what it is), a bounding box (where it is, as a rectangle), and a confidence score (how sure the model is). Classification answers 'Is there a cat in this photo?'; detection answers 'There is a cat HERE, a dog THERE, and a bicycle over THERE.' It localizes and names every object it can find, all at once.

Now look at the structural difference that makes detection a genuinely different kind of problem. A single photo might contain 0 objects (an empty sky), 1 object (a portrait), or 50 objects (a crowded street). So the output length is not fixed — it depends on the contents of the image, which the network does not know in advance. This one fact, a variable number of outputs, is the through-line of this entire track. Almost every clever idea you will meet — anchors, queries, non-maximum suppression — exists to wrestle with the problem of producing an unknown number of answers from a fixed-size network.

The bounding box: a rectangle with coordinates

The 'where' part of every detection is carried by a bounding box — an axis-aligned rectangle drawn tightly around an object. It is the fundamental geometric primitive of detection: the smallest upright rectangle that fully contains the object. 'Axis-aligned' just means its edges are horizontal and vertical (not tilted), which keeps the description simple — four numbers are enough to pin it down.

Before we write those four numbers, we must agree on the coordinate system, because it trips up almost every beginner. In images the origin (0,0) sits at the top-left corner. The x-axis increases to the right, exactly as you'd expect — but the y-axis increases downwards. This is the opposite of the math classroom, where y points up. So a pixel at (x=10, y=5) is near the top of the image, and increasing y moves you DOWN the page. Memorise this once and a hundred small bugs disappear.

The two standard parametrizations of a bounding box, and how to convert from corners to centre+size.

Let's unpack this completely. The corner form (x_1, y_1, x_2, y_2) names two opposite corners: (x_1, y_1) is the top-left corner and (x_2, y_2) is the bottom-right corner. The centre form (c_x, c_y, w, h) instead names the box's centre point (c_x, c_y) together with its width w and height h. The conversions read directly from geometry: the centre is the average of the two corners, so c_x is the midpoint between the left and right edges and c_y the midpoint between top and bottom; the width w = x_2 - x_1 is how far right edge sits from left edge, and the height h = y_2 - y_1 is how far the bottom edge sits below the top. Because x_2 is to the right of x_1 and (remember, y grows downward) y_2 is below y_1, both w and h come out positive — a negative width or height would mean the corners were swapped, which is a bug, not a valid box.

A concrete example. Say a face sits in a 640×480 image with top-left corner at (x_1, y_1) = (120, 80) and bottom-right corner at (x_2, y_2) = (200, 180) — that's the corner form. Converting: c_x = (120+200)/2 = 160, c_y = (80+180)/2 = 130, w = 200-120 = 80, h = 180-80 = 100. So the very same box in centre form is (160, 130, 80, 100): an 80-by-100 pixel rectangle centred at the point (160, 130). Both descriptions name the identical rectangle; they are just two languages for the same shape.

Why do networks usually predict the centre form rather than two corners? Two reasons. First, the four numbers become more independent and meaningful: 'where is it' (the centre) is cleanly separated from 'how big is it' (the size), and a network can place a box roughly right and then refine its size, which mirrors how detectors actually work. Predicting two arbitrary corners couples those concerns awkwardly. Second, sizes are naturally non-negative and often vary over a wide range, so they pair well with tricks (like predicting \log w) that keep training stable — a topic for later guides.

One more distinction: pixel coordinates versus normalized coordinates. So far our numbers were raw pixels (160, 130, …), which are tied to one specific image size. It is often better to divide every coordinate by the image's width or height, mapping everything into the range [0,1]: our box becomes c_x = 160/640 = 0.25, c_y = 130/480 \approx 0.27, w = 80/640 = 0.125, h = 100/480 \approx 0.21. Normalization means '40% of the way across' instead of '160 pixels in', so the same description survives when you resize the image — and resizing is something networks do constantly. It also keeps all four targets on a similar, small scale, which makes them easier to learn.

Two jobs at once: localization and classification

Look closely at one detection — a single (class, box, confidence) triple — and you'll see it is really two predictions stitched together. The first is localization: figure out the box coordinates. Since coordinates are continuous numbers, this is a regression problem — the model outputs real values like c_x = 0.25 and tries to make them match the true rectangle as closely as possible. The second is classification: given the contents of that box, decide WHAT object it is, choosing among the known categories. That part is exactly the classification you already know, just applied to a region rather than the whole image.

There is a third, quieter output that detection needs but classification never did: a confidence, often called an objectness score. It answers a yes/no-ish question: 'How sure am I that there is a real object here at all, rather than empty background?' This matters precisely because of the variable-output problem from Section 1. A detector typically proposes many candidate boxes and must be able to say 'this one is junk, ignore it' for the vast majority that landed on background. The confidence score is the dial it turns to do that. Often the model fuses objectness with the class probability into one number per box, but conceptually it is asking both 'is something here?' and 'what is it?'.

The network component that emits these outputs is called a detection head. Think of the network in two parts: a body that looks at the image and distils it into rich features, and a head that reads those shared features and produces, for each location it considers, the box coordinates AND the class scores AND the objectness. 'Head' is the standard name for the small output module bolted onto shared features; here it has two prongs (a regression prong for the box, a classification prong for the label) drawing on the same underlying representation, which is efficient — the hard visual work is done once and reused for both jobs.

Hold on to this, because it is the unifying secret of the whole track: every detector you are about to study — the slow-but-careful two-stage R-CNN family in guide 3, the fast one-stage YOLO and SSD in guide 4, and the anchor-free and transformer detectors in guide 5 — looks wildly different on the outside, yet they all ultimately emit the same thing: boxes, class labels, and confidences from a detection head. Object detection architectures differ in HOW they generate and refine candidate bounding boxes, not in what the final answer is shaped like.

Why detection is hard: the variable-output problem

It is worth cataloguing exactly why detection is so much harder than classification, because every piece of machinery in this track is a direct response to one of these difficulties. If you understand the problems now, the solutions later will feel inevitable rather than arbitrary. Here is the analogy to hold in mind: detection is like being handed a paper form with a fixed number of blank lines and being told to list every item in a photo — but the photo might contain three items or thirty, so a fixed-size form simply cannot fit the task. The mismatch between a fixed-size network and a variable-size answer is the source of nearly all the pain.

1. Unknown and variable object count. The image might hold 0, 1, or 50 objects, and the model does not know the number ahead of time. A network that outputs a fixed-size tensor must somehow represent an answer whose true length it cannot predict.
2. Huge scale variation. The same class can appear tiny or enormous — a person might be 20 pixels tall in the background or 2000 pixels tall up close — and the model must detect both, even though they look very different at the pixel level.
3. Occlusion and crowding. Objects overlap, hide behind one another, or pack tightly together, so the model must infer whole objects from partial, tangled evidence and still separate neighbours that touch.
4. Extreme class imbalance. The overwhelming majority of any image is background, not objects. If you scatter candidate boxes across an image, almost all of them land on empty regions, so a naive model can score well by lazily predicting 'background' everywhere.
5. The matching / assignment problem. When the model emits many candidate boxes, we must decide which candidate is 'responsible' for which real object during training — and which candidates are duplicates to be discarded at inference. Getting this bookkeeping right is subtle and central.

Here is the reassuring part: every single one of these challenges has a named answer that a later guide will teach in full. The variable object count is handled by laying down a fixed grid of reference rectangles called anchor boxes (or, in the newest models, a fixed set of learned 'object queries'), turning 'how many?' into 'which of these fixed slots fired?'. Scale variation is tackled by feature pyramids (FPN), which look for big and small objects at different resolutions. Occlusion-driven duplicate boxes are pruned by non-maximum suppression. The matching problem is solved using an overlap measure called IoU. And class imbalance is countered by focal loss, which stops easy background examples from drowning out the rare real objects. Do not worry about the details yet — just register that each pain has a planned cure.

The shape of a modern detector

Let's assemble a bird's-eye view of a modern detector — a skeleton the rest of this track will flesh out. It reuses two ideas you already know from earlier CNN tracks. First, a backbone convolutional network slides learned filters over the image and produces one or more feature maps: grids of feature vectors where each cell summarizes a patch of the original image. Second, each cell's view of the input is its receptive field — the region of the original image it can actually 'see'. A deep cell sees a large region, which is exactly what lets a single cell decide 'there is an object centred near me'.

1. Input image. A raw photo (say 640×480×3) enters the network.
2. Backbone CNN → feature map(s). The backbone distils the image into a compact grid of rich features, often at several resolutions so both small and large objects are represented.
3. Detection head → MANY candidate boxes. The head reads the feature map and emits a large pile of candidate boxes, each with class scores and a confidence — far more candidates than there are real objects.
4. Post-processing (NMS) → keep the best. Non-maximum suppression discards low-confidence and duplicate boxes, keeping one clean, non-overlapping box per object.
5. Final detections. What survives is the answer: a variable-length list of (label, box, confidence) triples.

# Bird's-eye pseudocode of a modern detector's forward pass.
# Every architecture in this track is a variation on these five lines.

def detect(image):
    features = backbone(image)          # CNN -> feature map(s)
    candidates = head(features)         # MANY boxes, each: (box, class_scores, objectness)

    # Turn raw outputs into scored detections
    dets = []
    for box, class_scores, objectness in candidates:
        label = argmax(class_scores)    # WHAT: pick the best class
        score = objectness * max(class_scores)  # confidence: object AND class
        if score > CONF_THRESHOLD:      # drop near-empty background boxes
            dets.append((label, box, score))

    # Remove duplicate boxes covering the same object (guide 2 explains NMS)
    final = non_max_suppression(dets, iou_threshold=0.5)
    return final                        # variable-length list of detections

Now you have a scaffold, and the rest of the track simply fills in the boxes of this pipeline. Guide 2 zooms into the post-processing and evaluation stage — how we measure whether a predicted box is good (IoU), how non-maximum suppression removes duplicates, and how mean Average Precision (mAP) scores a whole detector. Guide 3 opens up the head for two-stage detectors, the careful R-CNN → Faster R-CNN lineage. Guide 4 covers one-stage detectors (YOLO, SSD) that produce boxes in a single sweep, plus the focal loss that tames background imbalance. Guide 5 reaches the modern frontier: anchor-free detectors and the transformer turn, which rethink even the candidate-generation step. Through all of it, keep your eye on the through-line — how each design answers the variable-output problem of object detection.