cross-entropy loss (LLM)

Cross-entropy loss is the single number that tells the model how wrong its next-word guess was. At each position the model outputs a probability for every possible next token; cross-entropy looks only at the probability it gave to the token that actually came next, and penalises it by the negative logarithm of that probability. Guess the truth with high confidence and the penalty is tiny; assign the truth a low probability and the penalty is large.

Averaged over a whole batch, this loss is exactly what gradient descent pushes downward — its gradients tell every weight which way to move to make true continuations more likely. It also has a clean interpretation: cross-entropy measured in bits is the average surprise of the data under the model, and perplexity is simply its exponential, the model's effective number of equally likely choices per token.

Because the logarithm punishes confident wrong answers brutally, the loss strongly discourages the model from being certain about tokens it cannot predict. Watching this number fall smoothly over training is the heartbeat of a healthy pretraining run.