label smoothing

Label smoothing is a regularisation technique that softens the training targets. Instead of telling the network the correct class has probability 1 and every other class probability 0, which is a hard one-hot target, it assigns the correct class a probability of 1 minus epsilon and spreads the remaining epsilon uniformly across the other classes. A typical value is epsilon equal to 0.1, giving the true class 0.9 and sharing 0.1 among the rest.

The motivation is that hard targets push the network to make the correct logit infinitely larger than the others, encouraging overconfident, peaky outputs that generalise worse and are poorly calibrated. By making the target slightly soft, label smoothing penalises extreme logit gaps, which acts as a regulariser, tends to improve top-1 accuracy on large benchmarks like ImageNet, and notably improves calibration so predicted probabilities better reflect true correctness rates.

It is essentially free to implement because only the target vector changes while the cross-entropy loss is unchanged, which is why it appears in most modern training recipes. The main caveat is that the tighter, more uniform feature clusters it induces can slightly hurt downstream uses that rely on the geometry of the penultimate features, such as knowledge distillation, where a teacher trained with heavy label smoothing can transfer less information.