Datasets and Scorecards: How We Know a Classifier Is Good

Why the field needs shared benchmarks

In the last guide you built a classifier that turns pixels into a label. Naturally the next question is: is it any good? You could answer with a number — "94% correct!" — but that number is meaningless on its own. 94% on which images? Graded how? If every research lab tests on its own private pile of photos, then each lab is writing its own exam, marking its own paper, and announcing its own grade. No two scores can be compared, and nobody can tell whether a new method is genuinely better or just got an easier test.

The fix is the same one schools landed on: a standardised test that everyone sits. If thousands of students take the identical exam under the identical rules, then a higher score really does mean a stronger student. In computer vision the equivalent is a shared benchmark image dataset — a fixed collection of images that the whole community agrees to evaluate on. When everyone reports numbers on the same data, a leaderboard becomes a real ranking instead of a pile of incomparable boasts.

Two things flow from this discipline. First, reproducibility: because the data and splits are frozen, anyone can re-run your evaluation and get the same answer, so claims can be checked rather than trusted. Second, a level playing field: with the test held constant, the only way to move your score is to build a genuinely better model. That is exactly what turned vision from a collection of anecdotes into a measurable science — and, as we will see next, it is what let one dataset reshape the entire field.

A tour of the classics: MNIST, CIFAR, ImageNet

Benchmarks come in difficulty levels, like a video game. Climbing the ladder of canonical datasets gives you a mental map of the whole field, so let us walk up it from easiest to hardest. Each rung trades off the same two things: how fast you can iterate versus how realistic the task is.

MNIST is the "hello world" of vision: 70,000 handwritten digits (0–9), each a tiny 28×28 grayscale image. With only 10 classes and almost no clutter, you can train a decent model in seconds on a laptop. That makes MNIST perfect for sanity-checking code — but it is also so easy that good models pass 99%, leaving almost no room to distinguish a great method from a merely good one. The dataset is saturated.

CIFAR-10 and CIFAR-100 raise the bar. These are 32×32 colour photos of real things — CIFAR-10 has 10 everyday categories (airplane, cat, ship…), CIFAR-100 splits the same idea into 100 finer ones. The images are still tiny and fast to train on, but now there is real-world variation: lighting, pose, background. CIFAR is the classic mid-size playground where you can run many experiments a day yet still learn something meaningful.

At the top sits ImageNet: over a million full-resolution photos spread across 1000 fine-grained categories — not just "dog" but dozens of specific breeds, not just "snake" but many species. This scale and difficulty made it the gold-standard benchmark for image classification. The categories are drawn from real concepts, the photos are messy and natural, and 1000 classes means the task is genuinely hard. ImageNet will be our running example for every metric in this guide.

ImageNet is also a piece of history. For years progress on it was slow and incremental. Then in 2012 a deep convolutional network ("AlexNet") cut the error rate dramatically in a single leap — a result so far ahead of everything else that it convinced the field deep learning was the way forward. Almost the entire modern era of computer vision dates from that moment, and it happened on a benchmark precisely because the benchmark made the leap measurable and undeniable.

Train, validation, test: the cardinal rule

Every serious benchmark splits its images into three disjoint pools, and the discipline around them is the single most important habit in the whole field. The training set is what the model learns from. The validation set is what you use to make choices — which architecture, how much data augmentation, what learning rate. The test set is touched exactly once, at the very end, to estimate how well the model will do on data it has truly never seen.

The exam analogy nails why this works. The training set is your textbook and homework — study it as hard as you like. The validation set is the practice exam you take to decide what to revise. The test set is the final exam: you must never study from it, because the whole point is to predict how you will do on questions you have not seen. If you have already seen the final, your grade tells you nothing about real understanding.

1. Train on the training set: fit the model's parameters until it learns the task.
2. Tune on the validation set: try different architectures, augmentations, and learning rates; keep whatever scores best here. Repeat as often as you like.
3. Test ONCE on the test set: with all choices frozen, run the model a single time and report that number as your true generalisation estimate.

You already met overfitting in earlier machine-learning tracks: a model can memorise its training data — scoring nearly perfectly there while doing badly on anything new. The validation set is how you catch this. As the figure shows, when training error keeps dropping but validation error starts climbing, the model has stopped learning the general pattern and started memorising. That gap is your early-warning signal to stop, simplify, or add regularisation.

There is a subtler trap. You can also overfit to the test set — not by training on it, but by peeking at it again and again. Each time you tweak your model, check the test score, and tweak again based on what you saw, you are quietly using the test set to make decisions. After hundreds of such peeks you have effectively fitted to it, a habit known as benchmark hill-climbing. Public leaderboards make this worse: when thousands of teams all repeatedly evaluate against the same hidden test set, the community as a whole slowly "leaks" the answers, and reported numbers drift higher than true generalisation. That is why guarding the test set — and treating its single use as sacred — is the cardinal rule.

Top-1 and top-5 accuracy

Start with the simplest metric of all: plain accuracy is the fraction of test images the model gets exactly right. Recall from Guide 1 that the model outputs a probability for each class, and the decision rule (argmax prediction) picks the single class with the highest probability. An image counts as correct when that top pick equals the true label. So ordinary accuracy is also called top-1 accuracy: top-1 = argmax. It is the strictest fair test — the one guess has to be right.

We can generalise this. Instead of asking only about the single best guess, top-k accuracy asks: is the true class among the model's k highest-probability guesses? You read off the k most confident classes and count the image as correct if the right answer is anywhere in that short list. Here is the definition.

The fraction of images whose true class appears among the model's k most probable predictions.

Let us unpack every symbol. N is the number of test images, and n indexes them (image 1, image 2, …, up to N). For image n, the model produces a probability vector p_n with one entry per class; y_n is the true class of that image. TopK(p_n) is the set of the k classes with the highest entries in p_n — the model's k best guesses. The square brackets 1[ … ] are the indicator function: it equals 1 when the statement inside is true (the true class is in the top-k set) and 0 when it is false. Summing those 1s and 0s over all N images and dividing by N gives the fraction that landed inside the top-k — i.e. the accuracy. Notice that when k = 1, TopK is just the single argmax class, so the formula collapses back to ordinary top-1 accuracy.

Why did top-5 become standard on ImageNet? Because with 1000 fine-grained classes, a single guess is a brutally harsh bar. Many of those classes are genuinely ambiguous or overlapping: there are dozens of dog breeds that even humans confuse, and a single photo may legitimately contain several objects (a person riding a horse, a desk covered in supplies). Demanding that the model's one top guess match the one chosen label can punish answers that are basically reasonable. Top-5 asks the gentler, often more meaningful question: is the right answer at least in the running?

A concrete example makes the difference vivid. Suppose the true class of a photo is "Siberian husky." The model's ranked guesses are: 1st "Alaskan malamute" (0.30), 2nd "Eskimo dog" (0.25), 3rd "Siberian husky" (0.20), 4th "wolf" (0.15), 5th "Samoyed" (0.10). Under top-1 the prediction is "malamute", which is wrong — this image scores 0. But the true class sits at rank 3, comfortably inside the top 5, so under top-5 it scores 1. Same model, same photo: harsh by one metric, correct by the other. Reporting both numbers tells a fuller story.

import numpy as np

def top_k_accuracy(probs, labels, k=5):
    # probs:  array of shape (N, C) - predicted probability of each class
    # labels: array of shape (N,)   - the true class index for each image
    # returns the fraction of images whose true class is among the top-k guesses
    correct = 0
    for p, y in zip(probs, labels):
        topk = np.argsort(p)[-k:]   # indices of the k largest probabilities
        if y in topk:               # is the true class in the short list?
            correct += 1
    return correct / len(labels)

# k=1 reproduces ordinary accuracy: the single argmax guess must equal the label.
# k=5 is the classic ImageNet metric.

Reading a confusion matrix

A single accuracy number tells you HOW good a model is, but not WHAT it is good and bad at. To get that diagnosis we use a confusion matrix. The idea is simple: lay out a grid where rows are the true class and columns are the predicted class. Each cell counts how often an image of the row's class was predicted to be the column's class. Add up every cell and you get the total number of test images.

The diagonal cells — where the true class equals the predicted class — are the correct predictions. So a perfect model puts every count on the diagonal and zero everywhere else. Every off-diagonal cell is a mistake, and crucially it tells you exactly which mistake: a count in row "cat", column "lynx" means cats that were called lynxes. A bright off-diagonal cell reveals a pair of classes the model systematically confuses — which is precisely the information you need to know where to improve.

Predicted ->     cat    dog   lynx     (row = true class)
  cat            7      0      3     -> 10 cat images
  dog            0     10      0     -> 10 dog images
  lynx           4      0      6     -> 10 lynx images

# Diagonal (correct): 7 + 10 + 6 = 23  ->  accuracy = 23/30 = 0.77
# Dogs are perfect. The bright off-diagonal cells (cat<->lynx)
# show the model keeps swapping cats and lynxes: 3 cats called lynx,
# 4 lynxes called cat.

Read this small example like a doctor reading an X-ray. Overall accuracy is 23/30 ≈ 77% — fine, but uninformative on its own. The matrix tells the real story: the "dog" row is perfect (10 on the diagonal, zeros elsewhere), so dogs are not the problem. The damage is concentrated in the cat–lynx block: 3 cats were called lynxes and 4 lynxes were called cats. That makes sense — cats and lynxes look alike — and it points to a concrete fix: get more or better cat-vs-lynx examples, or features that separate them. This is the bridge from "how good overall" to "good at what, bad at what," the same task of image classification viewed through a diagnostic lens.

Beyond one number: precision, recall, and per-class views

Accuracy has a dangerous blind spot: class imbalance. Imagine a medical screening dataset of 1000 images where only 50 patients are actually sick and 950 are healthy. A lazy model that ignores its input and always shouts "healthy!" gets 950 of 1000 right — 95% accuracy — while catching exactly zero sick patients. The number looks brilliant and the model is useless. When one class dominates, even top-k accuracy can flatter a model that has learned nothing about the cases you care about.

To see through this we split correctness into two complementary questions. Precision asks: of everything I flagged as sick, how much was actually sick? Recall asks: of everyone who was truly sick, how many did I catch? Suppose a better model flags 60 patients as sick; of those, 40 really are sick (true positives) and 20 are false alarms (false positives), while 10 sick patients are missed (false negatives). Then precision = 40/60 ≈ 0.67 (two-thirds of alarms were real) and recall = 40/50 = 0.80 (we caught four-fifths of the sick). Two numbers, two very different things — and neither is the misleading 95%.

Precision and recall pull against each other, and the knob between them is the decision threshold. Recall from Guide 1 that the model outputs a probability; we declare "sick" when that probability exceeds some threshold. Lower the threshold and you flag more patients: recall rises (you miss fewer sick people) but precision falls (more false alarms). Raise the threshold and the reverse happens: precision rises, recall drops. There is no free lunch — you choose where to sit on this trade-off based on what is costlier in your application. In cancer screening a missed case is catastrophic, so you favour recall; for a spam filter a wrongly-deleted email is annoying, so you favour precision.

The same fix scales up to many classes. Instead of one overall accuracy, compute the accuracy (or precision and recall) for each class separately — a per-class view — and then average those per-class scores to get balanced accuracy. The difference is decisive under imbalance: ordinary accuracy weights every image equally, so big classes dominate, while balanced accuracy weights every class equally, so a model that ignores rare classes can no longer hide. The 95%-by-doing-nothing model would score a damning 50% balanced accuracy (perfect on healthy, zero on sick).