loss function

A loss function is the single number that tells a network how badly it is doing on a given example or batch, the quantity training tries to make small. It compares the network's prediction to the desired answer and returns a scalar: zero (or minimal) when the prediction is perfect, larger when it is worse. Training is literally the search for parameters that minimize the average loss over the data, so choosing the loss is choosing what good means for your task.

A loss L(y_hat, y) maps a prediction y_hat and a target y to a non-negative scalar (lower is better); the training objective is the empirical risk, the average loss over the training set, often plus a regularization term. For the gradient-based learning that backpropagation enables, the loss must be differentiable almost everywhere with respect to the predictions. The loss also encodes assumptions: many standard losses are negative log-likelihoods under a probabilistic model (squared error corresponds to Gaussian noise, cross-entropy to a categorical or Bernoulli model), so choosing a loss is implicitly choosing a noise model.

Different vision tasks demand different losses. Classification uses cross-entropy. Regression of continuous quantities (depth, keypoint coordinates, bounding-box offsets) uses squared error (L2), absolute error (L1), or the robust Huber or smooth-L1 that resists outliers. Segmentation often adds Dice or IoU losses that directly target overlap. Detection mixes a classification loss with a box-regression loss such as GIoU. Generative and self-supervised models use specialized objectives: the adversarial loss of GANs, the denoising score-matching loss of diffusion models, the InfoNCE contrastive loss behind CLIP, and perceptual losses comparing deep features rather than raw pixels.

A crucial subtlety: the loss is what you optimize, but it is usually not the metric you ultimately care about (accuracy, mAP, IoU, FID). Many true metrics are non-differentiable or flat, so they cannot be optimized directly; the loss is a differentiable, well-behaved surrogate chosen to correlate with the metric while admitting gradients. A model can lower its loss while a metric stagnates, which is why you monitor both.