How Good Is My Model, Really? Accuracy, Confusion, Precision and Recall

What does "good" even mean?

A model lands on your desk with one proud sentence: "It's 95% accurate." Before you celebrate, ask the question this entire track is built around — good for what? A 95% number that is fine for sorting holiday photos could be a disaster for screening medical scans, where the 5% it gets wrong might be exactly the tumors. The same number that unlocks your phone reliably might wave a stranger straight in. The headline figure hasn't lied to you, but it hasn't told you the whole truth either. Learning to read past it is a real skill, not a footnote you tack on after the model is trained.

Let's anchor everything in one concrete example we'll reuse for the whole guide: a binary classifier that looks at an image and says cat or not-cat. We always judge it on a test set — images it never saw during learning — because grading it on the training set it memorized would tell us nothing about new photos (you met this split earlier in the ladder; here we just lean on it). The simplest score is classification accuracy: the fraction of test images it labels correctly. It feels like the obvious measure of "good." By the end of this guide you'll see exactly when it earns your trust and when it quietly betrays you.

The confusion matrix: four boxes that explain everything

Almost every metric you'll meet — accuracy, precision, recall, the ROC curve — is computed from one small table called the confusion matrix. If you truly understand this table, the rest is just arithmetic on top of it. For our two-class cat detector it's a 2×2 grid that cross-tabulates two things: what the model said, and what was actually true. Every test image falls into exactly one of the four boxes.

The four boxes have names that sound intimidating but follow one tidy rule. The word positive or negative describes what the model said (positive = "cat", negative = "not-cat"); the word true or false describes whether the model was right. So: a true positive (TP) is a real cat the model called a cat; a false positive (FP) is a not-cat the model wrongly called a cat (a false alarm); a true negative (TN) is a not-cat correctly called not-cat; a false negative (FN) is a real cat the model missed and called not-cat. Memory hook: read the second word for what the model claimed, and the first word for whether to believe it.

1. Start with 100 test images. Suppose 20 are really cats and 80 are really not-cats — that's the ground truth, fixed before the model says anything.
2. Run the model and compare each call to the truth. Of the 20 real cats it correctly flags 16 → TP = 16, and misses 4 → FN = 4 (4 cats slipped by, labelled not-cat).
3. Of the 80 real not-cats, it wrongly shouts "cat!" on 10 → FP = 10 (ten false alarms), and correctly says not-cat on 70 → TN = 70.
4. Sanity-check the grid: TP + FN = 16 + 4 = 20 real cats, and FP + TN = 10 + 70 = 80 real not-cats, totalling 100. Every image is accounted for exactly once.

Accuracy and its hidden trap

Now we can define accuracy precisely instead of hand-waving. Accuracy is simply the share of predictions that were correct — the two "right" boxes (TP and TN) divided by all four boxes. It's intuitive and often useful. But it carries a trap so common and so dangerous that spotting it is the single most important lesson of this whole guide.

Accuracy = correct calls divided by all calls.

Read it term by term. The numerator TP + TN counts every image the model got right — real cats called cat, plus real not-cats called not-cat. The denominator TP + TN + FP + FN is just the total number of test images (all four boxes summed). So accuracy is "how many did we get right, out of everything." On our running matrix: (TP + TN) / total = (16 + 70) / (16 + 70 + 10 + 4) = 86 / 100 = 0.86, i.e. 86% accurate. So far, so reasonable.

Now watch the trap spring. Imagine a different, imbalanced test set where cats are rare: out of 1,000 images, only 10 are cats (1%) and 990 are not. Build the laziest possible "model" — one that ignores the image entirely and always says not-cat. It never produces a single positive, so TP = 0 and FP = 0; it labels all 990 not-cats correctly (TN = 990) and misses all 10 cats (FN = 10).

Plug those into the formula: Accuracy = (0 + 990) / (0 + 990 + 0 + 10) = 990 / 1000 = 0.99. A useless model that has never once recognized a cat scores 99% — beating our genuinely-working 86% detector on the headline number! The deception comes from the denominator: it's dominated by 990 easy negatives, so getting the rare positives totally wrong barely dents the score. Look back at the confusion matrix and the fraud is obvious — TP = 0 means the model has zero ability to find the thing we actually care about, a fact the accuracy number cheerfully hides.

Precision and recall: two honest questions

The cure for the accuracy trap is to stop averaging over everything and instead ask two sharper, more honest questions about the positive class. These are precision and recall. Precision asks: "when the model shouts cat, how often is it actually right?" — it's about the trustworthiness of the model's alarms. Recall asks: "of all the real cats out there, how many did the model actually catch?" — it's about coverage, how little it lets slip by. The lazy always-not-cat model from before never shouts cat, so it has no precision to speak of and zero recall — instantly exposed.

Precision, recall, and the F1 score that balances them.

Let's decode each piece on our matrix (TP = 16, FP = 10, FN = 4). Precision = TP / (TP + FP) divides the cats it got right by everything it called a cat (right or wrong): 16 / (16 + 10) = 16/26 ≈ 0.62. So when this model yells "cat," it's correct only about 62% of the time. Recall = TP / (TP + FN) divides the cats it got right by all the cats that truly exist: 16 / (16 + 4) = 16/20 = 0.80. So it finds 80% of the real cats. Notice the same TP sits on top of both, but the denominators differ — precision is dragged down by false alarms (FP), recall is dragged down by misses (FN). F1 then squeezes these into one number: 2 · (0.62 · 0.80) / (0.62 + 0.80) = 2 · 0.492 / 1.415 ≈ 0.70.

# Counts read straight off the 2x2 confusion matrix
TP, FP, FN, TN = 16, 10, 4, 70

accuracy  = (TP + TN) / (TP + TN + FP + FN)              # 0.860
precision = TP / (TP + FP)                               # 0.615
recall    = TP / (TP + FN)                               # 0.800
f1        = 2 * precision * recall / (precision + recall)  # 0.696

print(f"accuracy={accuracy:.3f}  precision={precision:.3f}  "
      f"recall={recall:.3f}  f1={f1:.3f}")
# accuracy=0.860  precision=0.615  recall=0.800  f1=0.696

Why the harmonic mean in F1 instead of a plain average? Because the harmonic mean punishes lopsided scores. Picture a spam filter that, to never let any spam through, marks almost everything as spam: it achieves near-100% recall (it catches all the spam) but terrible precision (it also trashes your real email). A plain average of, say, recall 1.0 and precision 0.1 would give a comforting 0.55 — but F1 = 2·(1.0·0.1)/(1.0+0.1) ≈ 0.18, refusing to be fooled by the one good half. The lesson: there is no universally "best" metric. If a missed cat is the costly error (a tumor, a fraud), favour recall; if a false alarm is costly (flagging an innocent transaction), favour precision; F1 is the honest compromise when both matter.

Thresholds, the ROC curve, and AUC

Here's a secret hiding under everything so far: a classifier doesn't really blurt out a hard "cat / not-cat." Internally it produces a score — a number like 0.92 meaning "pretty sure it's a cat" or 0.13 meaning "probably not." We turn that score into a decision by picking a threshold: say, call it a cat if the score is above 0.5. Slide that threshold and the whole picture shifts. Lower it and the model says "cat" more eagerly — you catch more real cats (recall up) but also raise more false alarms (precision down). Raise it and the model gets stingy — fewer false alarms but more misses. There isn't one operating point; there's a whole curve of them, and this trade-off is exactly the precision–recall tension from the last section, now made continuous.

To picture the entire range of thresholds at once, we draw the ROC curve — the heart of the ROC curve and AUC. It plots two rates against each other as we sweep the threshold from strict to lenient. Before the formula, here's the intuition you'll need: each threshold gives the model one (false-alarm-rate, true-catch-rate) pair, i.e. one dot on the plot, and stringing all those dots together as the threshold slides traces out the curve.

The two axes of the ROC curve, and AUC as the area beneath it.

Decode the rates. TPR (true-positive rate) = TP / (TP + FN) is just recall under a new name — of all the real cats, the fraction caught; it's the vertical axis, and we want it high. FPR (false-positive rate) = FP / (FP + TN) is the fraction of real not-cats that got falsely flagged; it's the horizontal axis, and we want it low. On our worked matrix, TPR = 16/(16+4) = 0.80 and FPR = 10/(10+80... )— careful, FP+TN = 10+70 = 80, so FPR = 10/80 = 0.125. That single threshold gives the point (0.125, 0.80) on the ROC plot. Slide the threshold to a new value and you get a new (FPR, TPR) point; the collection of all of them is the curve.

Now AUC — the area under that ROC curve — and the interpretation beginners always miss. AUC has a beautiful plain-English meaning: it is the probability that the model gives a randomly chosen real cat a higher score than a randomly chosen non-cat. AUC = 1.0 means the model always ranks any cat above any non-cat (perfect separation); AUC = 0.5 means it's no better than a coin flip (the diagonal line). It's a threshold-free summary of how well the scores rank positives above negatives. One caution that ties straight back to the accuracy trap: on heavily imbalanced data, FPR has that huge pool of negatives in its denominator, so even many false alarms barely move it — ROC/AUC can look flatteringly good while precision is dismal. When positives are rare, prefer the precision–recall curve, which keeps the rare class front and centre.

A first hint that confidence can lie

Step back and notice what every tool in this guide measured: which class the model picks, and how often that choice is right. The confusion matrix, accuracy, precision, recall, ROC — all of them judge the model's decisions. But none of them asked a different, sneakier question: when the model attaches a confidence to a decision — "I'm 0.92 sure this is a cat" — does that 0.92 actually mean anything?

That question is calibration, the subject of model calibration. Here's the one intuition to plant for now, borrowed from weather forecasting: think of all the days a forecaster says "70% chance of rain." If the forecaster is well-calibrated, it should actually rain on about 70% of those days — no more, no less. A model can be a brilliant ranker (high AUC, picks the right class) yet wildly overconfident, slapping 0.99 on guesses that are right only 70% of the time. For a phone unlock or a medical decision, that gap between stated and actual confidence matters enormously — and it's invisible to every metric we've built so far.

That's the doorway into the rest of this track. Guide 2 pushes past accuracy into the richer metrics for detection, segmentation, generation, and exactly this confidence/calibration question. Later guides ask whether the model holds up when pixels are deliberately tampered with (adversarial examples), when the world it meets at deployment drifts away from its training data (distribution shift), and whether it's fair and interpretable enough to trust. This first guide is the foundation they all stand on.