When Classification Gets Hard: Fine-Grained, Imbalanced, and Structured Labels

Fine-grained classification: telling near-identical things apart

So far our image classification examples have been comfortably distinct: cat vs. dog vs. car. But many real problems ask you to separate categories that look almost the same. Fine-grained classification is the task of telling apart visually very similar subcategories within one broad type — not 'is this a bird?' but 'is this a Tennessee warbler or an Orange-crowned warbler?' The same flavour shows up in dog breeds, car models, and aircraft variants. The high-level shape is identical; only the details differ.

The core difficulty is a brutal mismatch between two kinds of variation. The inter-class difference (between two species) can be tiny — two warblers might differ only in a faint wing bar or a slightly yellower throat. Meanwhile the intra-class variation (within one species) is huge — the same bird appears in different poses, lighting, seasons, and backgrounds. So a photo of warbler A in shade can look more different from another photo of warbler A than from a photo of warbler B. The thing that actually separates the classes is small, localised, and easily drowned out.

This is why generic global features are not enough, and why the feature maps from earlier tracks become the hero. Recall that a convolutional network produces a spatial grid of features at each layer — each cell describes a small patch of the image. Fine-grained methods lean on exactly this localisation: they use higher-resolution inputs so tiny parts survive, and they add part-attention that learns to focus the score on the discriminative region (the wing, the beak, the headlight cluster) rather than averaging over the whole frame. Concretely: to call a bird an Orange-crowned warbler, a good model effectively zooms into the crown and throat, compares the localised features there, and ignores the branch it is sitting on. Hold onto this idea of 'looking at the right small part' — section 5's graceful degradation builds on it.

Multi-label: when an image has many truths

Guide 1 teased a fork in the road; here we take the other branch. Everything so far assumed exactly one correct label per image — that is single-label classification, the setting behind softmax and top-1 accuracy. But a street photo can legitimately contain a person, a dog, AND a frisbee all at once. In multi-label classification the classes are NOT mutually exclusive: several can be true simultaneously, and zero can also be true (an empty scene). This single change breaks softmax.

Why does softmax fail? Softmax forces the class probabilities to compete and to sum to 1 — give more to 'dog' and you must take some away from 'person'. That is exactly what you want when the classes fight over one slot, and exactly wrong when they coexist. The fix is to drop the competition: apply an independent sigmoid per class, turning each logit into its own yes/no probability, and train with binary cross-entropy summed over all classes — each class is its own little present-or-absent question.

Per-class sigmoid + binary cross-entropy, summed over classes.

Let us read this symbol by symbol. The index c runs over every class. z_c is the raw score (the logit) the network outputs for class c. The sigmoid 1/(1+e^{-z_c}) squashes that score into a probability p_c that lands strictly between 0 and 1 — and crucially it does so independently of every other class, so the p_c values need not sum to 1. y_c is the ground-truth label for class c: 1 if the class is present in the image, 0 if absent. Now the loss: when the class IS present (y_c = 1) the first term -log p_c is active and shrinks as p_c rises toward 1, so the model is rewarded for being confident it is there; when the class is ABSENT (y_c = 0) the second term -log(1-p_c) takes over and shrinks as p_c falls toward 0, rewarding confidence it is not there. Summing over c just adds up the verdict on every class. Quick number: if a present class has p_c = 0.9, its loss is -log(0.9) ≈ 0.105 (small, good); if the model wrongly gave it p_c = 0.1, the loss is -log(0.1) ≈ 2.303 (large, a strong push).

Worked example — an image truly labelled {person, dog, frisbee}. Suppose the sigmoids come out as person 0.88, dog 0.73, frisbee 0.60, cat 0.05, car 0.12. These are five independent yes/no probabilities, not a competition. To turn them into a prediction we apply a per-class threshold (say 0.5): everything at or above the threshold goes into the predicted SET, everything below is dropped. Here the prediction is {person, dog, frisbee} — exactly right. Note the output is a SET, not a single winner, and thresholds can differ per class (a rare-but-costly class might use a lower threshold to catch more of it). This is also why the target representation differs: instead of a one-hot vector with a single 1, multi-label uses a multi-hot binary vector that can carry several 1s — or none.

# Single-label (mutually exclusive): softmax forces classes to compete
probs = softmax(logits)          # values sum to 1 across classes
pred  = argmax(probs)            # exactly ONE label wins

# Multi-label (independent truths): sigmoid per class, no competition
probs = sigmoid(logits)          # each in (0,1); do NOT sum to 1
preds = [c for c in classes if probs[c] >= threshold[c]]  # a SET

# Target representation
one_hot   = [0, 0, 1, 0, 0]      # single-label: exactly one 1
multi_hot = [1, 0, 1, 1, 0]      # multi-label: any number of 1s

Because the answer is a set, evaluation also shifts. Top-1 accuracy is meaningless — there is no single 'top' truth. Instead we score each label separately with precision (of the things we predicted present, how many really were) and recall (of the things really present, how many we caught), and summarise across classes with mean average precision (mAP), which rewards ranking the right labels above the wrong ones at every threshold. Contrast this cleanly with single-label: there, the target is the one-hot label encoding — a vector that is all zeros except a single 1, encoding 'exactly one of these is true'. Multi-label simply relaxes that constraint, and every downstream choice (sigmoid, BCE, thresholds, mAP) follows from relaxing it.

Long-tailed recognition: the rich-get-richer problem

Textbook datasets are tidy: every class has roughly the same number of examples. The real world is not. Photograph whatever crosses your camera and you get thousands of pictures of common house sparrows and exactly four of some rare endemic finch. This is long-tailed / class-imbalanced classification: a few 'head' classes hold the vast majority of examples, while a long 'tail' of many classes each holds only a handful. Plotted by frequency, the curve has a tall head and a long, thin tail — hence the name.

Why does naive training fail the tail? Recall the training loss is the cross-entropy summed (averaged) over the whole dataset. If the dataset is head-dominated, the model can drive that average way down just by acing the common classes and barely ever predicting the rare ones — the tail contributes too few terms to matter. Worse, the optimisation literally bends the geometry toward the head: the decision boundaries and the per-class bias terms drift so the model defaults to head answers, because predicting 'house sparrow' is right most of the time. The tail gets squeezed into thin slivers of feature space, or ignored entirely.

This is precisely the warning from Guide 2 made concrete: under imbalance, overall accuracy lies. Imagine a bird app where the head species are recognised at 97% per-class accuracy but the tail species at 18%. Because the head supplies most of the test images, headline top-1 accuracy might read a glossy 93% — while a birder relying on it for the rare species is failed nearly five times out of six. The single number hides a yawning head-vs-tail per-class accuracy gap. You only see the gap if you measure per class.

Fixing the tail: re-sampling, re-weighting, and decoupling

There is a standard toolkit for long-tailed / class-imbalanced classification, and each tool attacks the same root cause — the head drowns out the tail — from a different angle. The first is re-sampling: change what each training batch looks like. Oversample the rare classes (show their few images more often) and/or undersample the common ones (show fewer of them) so that, batch by batch, the model sees a more balanced mix and the tail actually pushes back on the gradients. The catch: aggressive oversampling repeats the same rare images many times and can overfit them, while heavy undersampling throws away real head data — so it is usually tuned, not maxed out.

The second tool is re-weighting the loss: leave the data alone but make a tail mistake count for more than a head mistake, so the summed loss can no longer be minimised by ignoring rare classes. The naive choice is to weight each class by inverse frequency (1/n_c), but that is too aggressive: the 1000th photo of a head class teaches the model almost nothing new (the images overlap heavily), so treating 1000 examples as 1000× more valuable than 1 vastly overstates the head. The 'effective number of samples' idea fixes this: because of diminishing returns, the real information in n_c samples saturates, so we weight by an effective count rather than the raw count.

Class-balanced loss weighting via the effective number of samples.

Symbol by symbol: n_c is the number of training samples in class c; L_c is that class's ordinary loss (e.g. its cross-entropy contribution); and w_c is the weight we multiply it by. The hyperparameter β (beta) sits in [0,1) and controls how fast extra samples stop adding value — it is the 'diminishing returns' dial. Notice β^{n_c} shrinks toward 0 as n_c grows, so the denominator (1 − β^{n_c}) grows toward 1 for big head classes (small weight) but stays small for tiny tail classes (large weight) — exactly the boost the tail needs. The two limits make it click: as β→0 every w_c→(1−0)/(1−0)=1, i.e. no re-weighting; as β→1 it approaches plain inverse-frequency weighting. Worked example with β = 0.99: a head class with n_c = 1000 gets w ≈ (0.01)/(1 − 0.99^1000) ≈ 0.01/0.99996 ≈ 0.0100; a tail class with n_c = 10 gets w ≈ (0.01)/(1 − 0.99^10) ≈ 0.01/0.0956 ≈ 0.105. The tail is weighted about 10× more than the head — but note raw inverse frequency would have demanded 1000/10 = 100×, so the effective-number trick tempers a 100× over-correction down to a saner ~10×.

# Class-balanced re-weighting via 'effective number of samples'
beta = 0.99
for c in classes:
    n_c = count[c]                          # samples in class c
    eff_num = (1 - beta ** n_c) / (1 - beta)  # 'effective' count
    w[c] = 1.0 / eff_num                     # rare class -> larger weight
normalize(w)                                  # keep total loss scale stable
loss = sum(w[c] * cross_entropy_c for c in classes)

The third, and often most effective, tool is decoupling representation learning from the classifier. The surprising empirical finding is that good features are best learned on the data as it naturally is (imbalanced) — re-sampling can hurt the feature extractor — whereas the classifier (the final linear layer and its biases) is exactly what gets skewed toward the head. So you train in two stages: first learn the backbone on natural data, then freeze the features and re-balance ONLY the final classifier (e.g. retrain it with balanced sampling, or just rescale its per-class biases). You get strong features and a fair decision rule, without the overfitting cost of resampling everything end-to-end.

Hierarchical classification: labels that live in a tree

Real label sets are rarely a flat list; they are usually a taxonomy — a tree where specific categories nest inside broader ones. ImageNet itself is built on the WordNet tree: a husky is a kind of dog, a dog is a kind of mammal, a mammal is a kind of animal (husky ⊂ dog ⊂ mammal ⊂ animal). The labels carry built-in relationships, and ignoring those relationships throws away information that is sitting right there in the label space.

Hierarchical classification takes this tree seriously in two ways. First, predictions should respect the tree: if the model says 'husky' it implicitly commits to 'dog', 'mammal', and 'animal' too, and a sensible system never outputs a leaf that contradicts its own ancestors. Second — and this is the heart of it — not all errors are equal. Confusing a husky with a malamute (two leaves under 'dog') is a small, forgivable slip; confusing a dog with a vehicle is a catastrophic one. Flat image classification treats both as simply 'wrong', scoring them identically. The tree lets us say one mistake is far cheaper than the other.

1. Predict at multiple levels: give the model heads (outputs) for coarse, mid, and fine levels — e.g. animal-vs-vehicle, then dog-vs-cat, then husky-vs-malamute — and train each against its level's label.
2. Enforce consistency: penalise or forbid predictions where the fine answer disagrees with the coarse one (a 'husky' that the coarse head calls a vehicle), so coarse and fine predictions tell the same story.
3. Back off to a confident ancestor when unsure: if the model cannot decide between husky and malamute, let it answer the parent 'dog' with high confidence instead of guessing a leaf — a partially correct, useful answer rather than a coin-flip.

Choosing the right metric for hard problems

Every hard setting in this guide quietly breaks plain top-1 accuracy, so we close by matching the metric to the problem — the same lesson as Guide 2, now applied case by case. The rule of thumb: the metric must encode what you actually care about, because whatever you measure is what training and model selection will chase.

For long-tailed data, prefer balanced (mean-per-class) accuracy over plain top-k accuracy. Plain top-1 averages over images, so the head dominates; mean-per-class averages the accuracy of each class first and then averages those, giving every class — head or tail — an equal vote. The tail finally counts. For fine-grained problems, lean on top-5 (it forgives a near-miss between genuine look-alikes) and read the confusion matrix, which reveals exactly which species get swapped for which — the off-diagonal hot spots are your hard pairs.

For hierarchical labels, use hierarchy-aware metrics that give partial credit for a correct ancestor: answering 'dog' when the truth is 'husky' should score better than answering 'truck'. Common choices weight an error by the tree distance between prediction and truth, so confusing two breeds barely dents the score while confusing a category across the tree is punished hard. For multi-label, drop top-1 entirely and report per-label precision/recall plus mean average precision (mAP), which together describe how well the predicted SET matches the true set across thresholds.

A short worked contrast nails the point. Suppose head species reach 98% per-class accuracy and tail species only 20%, and the head supplies the bulk of the test images. Headline top-1 might land around 0.95×0.98 + 0.05×0.20 ≈ 94% — looks great. But balanced accuracy averages the classes equally: with roughly as many tail classes as head, mean-per-class ≈ (0.98 + 0.20)/2 ≈ 0.59. The same model is '94%' or '59%' depending only on which metric you trust, and the 59% is the one that matches the user who cares about rare species.