mixed-precision LLM training

Numbers inside a network can be stored with more or fewer bits. Full thirty-two-bit precision is accurate but heavy; mixed precision keeps most of the math in a compact sixteen-bit format (commonly bf16) and reserves full precision only where it really matters. The payoff is direct: roughly half the memory and much faster arithmetic on modern accelerators, which are built to crunch low-precision numbers many times faster.

The art is knowing where low precision is safe. Forward and backward passes run in bf16, but a master copy of the weights and the optimizer's accumulators are kept in higher precision so tiny updates don't vanish into rounding. bf16 is favoured over older fp16 because its wider exponent range rarely overflows. The newest hardware pushes further to eight-bit fp8 for the heaviest matrix multiplies, with careful scaling to stay numerically safe.

Done right, mixed precision is nearly free accuracy-wise and is standard for every large run. Done carelessly it bites: lost precision can underflow gradients to zero or let loss spikes slip through, so it pairs with loss scaling and vigilant monitoring.