Scoring the Guess: Logits, Softmax, and Loss Functions

From neuron outputs to class scores: logits

Let's pick up exactly where the last guide left off. We pushed an image through a stack of neurons and reached the very last layer — usually a fully connected layer that has one output neuron per class. If your task is to decide whether a picture is a cat, a dog, or a bird, that final layer hands back three numbers, one per class. Nothing more has happened to them yet; they are simply the weighted sums that fell out of the final layer.

These raw final-layer numbers have a name: logits. The single most important fact about logits is that they are unnormalized. A logit can be any real number — large positive, large negative, or zero. They are not probabilities: they do not lie between 0 and 1, and they do not add up to anything in particular. So far, only their relative ordering carries meaning — a bigger logit means the network leans more toward that class, but by how much is not yet expressed in any honest, comparable way.

Let's make it concrete. Suppose for one photo the final layer returns the logits [2.0, 0.5, -1.0] for [cat, dog, bird]. We can read off the ranking instantly: cat scores highest, then dog, then bird (which even went negative). But now ask the two questions a real system must answer: which class do we predict, and how confident are we? The first is easy — pick the largest logit, so 'cat'. The second is genuinely awkward. Is 2.0 'very confident'? Compared to what? The gap to dog is 1.5; is that a lot? There's no scale. Raw logits simply cannot tell us 'cat with 79% probability' — and that is exactly the number a downstream decision (or a doctor, or a self-driving car) needs.

Softmax: turning scores into probabilities

The standard tool for converting a vector of logits into honest probabilities is softmax. It takes the whole vector of logits at once and returns a new vector of the same length where every entry is positive and all entries add up to exactly 1 — in other words, a genuine probability distribution over the classes. Crucially it preserves the ranking (the biggest logit still becomes the biggest probability), but now the magnitudes mean something we can compare and report.

Softmax does its job in two intuitive moves. First, exponentiate every logit — raise e to the power of each one. Exponentiating fixes two problems at once: e to any power is always positive (so negative logits stop being a problem), and the exponential grows fast, so it amplifies the lead of the biggest logit. Second, normalize: add up all the exponentials and divide each one by that total. Dividing by the sum forces the results to add to 1, which is what makes them a probability distribution.

Softmax: exponentiate one logit, then divide by the sum of all the exponentials.

Let's read every symbol. The vector z holds all K logits, and z_i is the logit for the one class i we are scoring (for us, K = 3 classes). The numerator e^{z_i} is that logit's exponential — always positive, and larger logits give disproportionately larger values. The denominator, the big sum over j from 1 to K of e^{z_j}, simply adds up the exponentials of every class; it is the normalizer that guarantees the outputs sum to 1. The whole ratio is the probability the model assigns to class i: 'how big is my exponential compared to everyone's exponentials put together?'

import math

logits = [2.0, 0.5, -1.0]   # raw final-layer scores for [cat, dog, bird]

# Step 1: exponentiate every logit
exps = [math.exp(z) for z in logits]
# exps  -> [7.389, 1.649, 0.368]

# Step 2: normalize by the total so the result sums to 1
total = sum(exps)            # 7.389 + 1.649 + 0.368 = 9.406
probs = [e / total for e in exps]
# probs -> [0.786, 0.175, 0.039]

print(probs, "sum =", sum(probs))   # sum = 1.0

Follow the arithmetic on our example. Exponentiating gives e^{2.0} ≈ 7.389, e^{0.5} ≈ 1.649, and e^{-1.0} ≈ 0.368. Their total is about 9.406. Dividing each by that total yields 7.389/9.406 ≈ 0.786 for cat, 1.649/9.406 ≈ 0.175 for dog, and 0.368/9.406 ≈ 0.039 for bird. Notice they sum to 0.786 + 0.175 + 0.039 = 1.000, exactly as promised. Our vague logits [2.0, 0.5, -1.0] have become the clear statement 'cat 79%, dog 18%, bird 4%'. That is now a confidence we can actually use.

Why exponentiate rather than, say, just clip negatives to zero and divide? Two reasons that will matter enormously once we start learning. The exponential is smooth and differentiable everywhere, so the network can be nudged by tiny gradient steps (the subject of guide 4); a hard clip has a flat dead zone that kills those gradients. And exp rewards confidence multiplicatively — being one unit ahead in logit space multiplies your share by a factor of e (about 2.7x), which gives the model a sensible, scale-free way to express strong belief.

What a loss function actually is

We can now say what the network believes — 'cat 79%'. But was it right, and if not, how wrong? A loss function answers that with a single number: it measures the gap between the network's prediction and the true label. The rule is simple and unbending — lower is better, and zero means a perfect prediction. A confident, correct answer earns a tiny loss; a confident, wrong answer earns a big one.

Think of the loss as the network's report card — a single grade that summarizes how it did on a question. It matters far beyond just informing us, because the loss is the exact quantity the learning process (guide 4) will try to drive downward. The network has no other compass: it improves by changing its weights so that this one number gets smaller. Everything we choose here — softmax, then which loss — shapes what 'getting better' even means.

1. Predict: run the image forward through the network to get logits, then softmax to get class probabilities.
2. Measure: compare the prediction to the true label using the loss function to get a single number.
3. Adjust: nudge every weight a little in the direction that would have made the loss smaller (the job of guide 4).
4. Repeat: do this over and over across many examples until the loss stops falling.

One important distinction: the loss on a single example is one report card for one question. What we actually want to minimize is the average loss over the whole dataset — sometimes called the cost or the objective. Averaging matters because we want a network that does well across all our cats, dogs, and birds, not one that aces a single lucky image. Every formula in the next sections is written for one example; mentally, picture it averaged over thousands.

Cross-entropy loss for classification

Cross-entropy loss is the natural partner of softmax for classification. Its intuition is wonderfully direct: it looks only at the probability the network assigned to the correct class and asks 'how high was it?' If you put most of your probability on the right answer, your loss is tiny. As that probability slides toward zero, the loss climbs — gently at first, then brutally — because of a logarithm. In plain terms, cross-entropy punishes being wrong, and punishes being confidently wrong far more.

Cross-entropy loss for one example over K classes.

Symbol by symbol: y_i is the true label written as a one-hot vector — it is 1 for the single correct class and 0 for every other class. So for a true 'cat' label, y = [1, 0, 0]. p_i is the softmax probability the network gave class i. The sum runs over all K classes, but here is the elegant part: because y_i is 0 for every wrong class, all those terms vanish, and the sum collapses to just one survivor — the correct class, where y_i = 1. So the whole formula simplifies to L = -log(p_correct): the negative log of the probability you placed on the right answer.

Walk the numbers. Our network said cat with probability 0.786, and cat was the truth, so the loss is L = -log(0.786) ≈ 0.241 (using natural log) — small, as a good guess should be. Now imagine a bad day where the network gave the correct class only 0.05: the loss becomes L = -log(0.05) ≈ 3.00, more than twelve times larger. Push the correct-class probability to 0.99 and the loss is -log(0.99) ≈ 0.01, almost nothing; let it fall toward 0 and -log(p) shoots toward infinity. That is the shape we want: confident-and-right earns near-zero loss, confident-and-wrong earns an enormous one, which is exactly the pressure that teaches the network to be both correct and well-calibrated.

Mean squared error and when regression is the goal

Not every vision task is 'which class'. Sometimes the network must predict a continuous number — this is called regression, and it shows up constantly in vision: the four coordinates of a bounding box around an object, the depth (distance from the camera) of each pixel, the rotation angle of a face, the brightness of a denoised image. There is no notion of 'the correct class' here; there is a target value and a predicted value, and we care how far apart they are. The standard loss for this is mean squared error (MSE).

Mean squared error over n examples (or n predicted numbers).

Reading it: y_k is the true target value for item k (say, the real depth of a pixel in metres), and ŷ_k — 'y-hat' — is the value the network predicted for it. Their difference (ŷ_k - y_k) is the raw error, which can be positive or negative. Squaring it does two jobs: it makes every error positive (so over- and under-shooting both count as bad and can't cancel out), and it penalizes large errors disproportionately — a miss of 2 contributes 4, but a miss of 10 contributes 100. Finally (1/n) sums over the n numbers and averages, giving one tidy score. Tiny example: predictions [3.0, 5.0] versus targets [2.0, 5.0] give errors 1 and 0, squared 1 and 0, averaged (1+0)/2 = 0.5.

Hold MSE and cross-entropy side by side so you never confuse them. Cross-entropy answers 'which class': it scores a probability distribution against a one-hot label, and it lives downstream of softmax. MSE answers 'how far off is a number': it scores a predicted value against a target value, and it lives downstream of a plain linear output (no softmax). One measures wrong-ness of a choice; the other measures wrong-ness of a magnitude. The squared term is also why a single huge mistake can dominate the MSE — one wildly wrong depth pixel can outweigh hundreds of nearly-perfect ones.

Sigmoid for binary and multi-label problems

We've met sigmoid activation before as a squashing function inside neurons. Here it returns in a new role: as the output that produces a probability. Recall what softmax assumes — exactly one class is true, so the probabilities must sum to 1 and compete. That assumption is plainly wrong in two common situations: a simple yes/no (binary) question, and multi-label problems where each tag can be present independently. A single photo can genuinely be both 'beach' AND 'sunset' AND 'people'; forcing those to sum to 1 would make them fight over a budget they should not have to share.

The fix is to put an independent sigmoid on each output neuron. Each sigmoid squashes its own logit into its own probability between 0 and 1, completely ignoring the others — so 'beach' can be 0.95 while 'sunset' is also 0.90 and 'snow' is 0.02, with no requirement that they add to anything. Each output is now its own yes/no question. And the loss that pairs with a single sigmoid output is binary cross-entropy, the two-class cousin of the cross-entropy loss we built earlier — applied once per label and then summed or averaged across all the labels.

Binary cross-entropy for one label, with p the sigmoid of that label's logit.

Unpack it for one label. y is the true answer for this label, either 1 (the tag is present) or 0 (absent). p = σ(z) is the sigmoid of that label's logit z — the model's predicted probability that the tag is present. The formula has two terms and exactly one of them is alive at a time. When y = 1, the (1-y) factor is 0, leaving -log(p): this punishes a low p when the tag truly is there. When y = 0, the y factor is 0, leaving -log(1-p): this punishes a high p when the tag truly is absent. For example, if the true label is 1 ('sunset present') and the model said p = 0.90, the loss is -log(0.90) ≈ 0.105 — small and appropriate. If it had said p = 0.10 for that same present tag, the loss is -log(0.10) ≈ 2.30, much larger.

This binary form is not a separate idea — it is exactly the cross-entropy from before specialised to two outcomes (present vs absent). And it completes our output-layer toolkit. A handy way to remember the whole story: softmax means 'pick exactly one' (mutually exclusive classes, probabilities competing to sum to 1, scored by cross-entropy), while independent sigmoids mean 'each label answers yes/no on its own' (multi-label, probabilities free of one another, scored by binary cross-entropy). With logits, softmax, sigmoid, and the matching losses now in hand, we have everything needed to turn a guess into a single number that says how wrong it was — and in the next guide we will finally use that number to learn, by tracing it backward through the network with backpropagation.