cross-entropy loss
Cross-entropy loss measures how far a predicted probability distribution is from the true answer, and it is the standard loss for classification. Intuitively it asks: what probability did the model assign to the correct class? It then punishes the model in proportion to how low that probability was. Confidently right costs almost nothing; confidently wrong costs a lot. Because the penalty for a near-zero probability on the true class grows without bound (negative log p goes to infinity), cross-entropy strongly discourages the model from being confidently mistaken.
For a single example with true class given by a one-hot vector y (a 1 in the correct slot, 0 elsewhere) and predicted distribution p, cross-entropy is H(y, p) = negative of the sum over k of y_k·log(p_k), which reduces to negative log(p_correct), the negative log-probability of the true class. It comes from information theory (the expected extra bits to encode samples from y using a code optimized for p) and equals the negative log-likelihood of the labels under the model, so minimizing it is maximum-likelihood estimation. Averaged over a dataset it is the training objective for classifiers.
Cross-entropy is almost always applied to softmax outputs, and the pair is special: the gradient of softmax-cross-entropy with respect to the logits is simply p minus y, the predicted distribution minus the one-hot target. This clean form has no vanishing factor and is numerically stable, which is why frameworks fuse them into one operation (CrossEntropyLoss, softmax_cross_entropy_with_logits) that takes raw logits rather than probabilities. The binary case pairs sigmoid with binary cross-entropy, used for yes/no and multi-label tasks.
Cross-entropy trains essentially every image classifier (ImageNet ResNets, ViTs) and, applied per pixel, semantic segmentation networks. Common refinements: label smoothing replaces the hard one-hot with a slightly softened target to curb over-confidence and improve calibration; focal loss reshapes cross-entropy to focus on hard, rare examples in detection; and class weighting counters imbalance. The contrastive objective in CLIP is itself a cross-entropy over image-text similarity scores.
Three classes, true label = class 1, predicted p=(0.7, 0.2, 0.1): loss = negative log(0.7) which is about 0.357. If the model were less sure, p=(0.4, 0.3, 0.3): loss = negative log(0.4) which is about 0.916. Confident and correct is cheaper.
Pitfall: feed logits, not probabilities, to a fused cross-entropy, and never apply softmax beforehand (a double softmax silently degrades training). Also guard against log(0): always work in the log-domain (log-softmax) rather than taking the log of a separately computed softmax, which can produce negative infinity.