mixed-precision training
/ mikst-pruh-SIZH-un TRAY-ning /
Mixed-precision training means doing most of the number-crunching with lower-precision numbers — fewer decimal digits each — to go faster and use less memory, while keeping a few critical parts in full precision so accuracy doesn't crumble. Think of a carpenter who measures rough cuts to the nearest centimeter to work quickly, but switches to a millimeter ruler for the joints that actually matter. Coarse where it's safe, fine where it counts.
Computers store numbers with a fixed number of bits; more bits mean more exact values but more memory and slower arithmetic. Traditionally training used 32-bit numbers throughout. Mixed precision does the bulk of the work in 16-bit numbers (formats called FP16 or BF16), which the accelerator chips can chew through roughly twice as fast and which take half the memory. The danger is that very small numbers can round away to zero and break the learning; so the model keeps a 32-bit 'master copy' of its parameters and uses tricks like 'loss scaling' to protect the delicate updates.
Why it matters: this is one of the cheapest big wins in modern training — often nearly doubling speed and halving memory for little or no loss in final quality, which is why it is now standard practice. The honesty caveat: it is not free magic. Done carelessly it can make training unstable or silently hurt accuracy, and not every operation tolerates low precision. It is a careful tradeoff that engineers tune, not a switch you flip and forget.
A layer's matrix multiply runs in fast 16-bit math, but the running tally of weight updates is kept in a 32-bit master copy. Tiny gradients that would round to zero in 16-bit survive in the 32-bit copy, so the model keeps learning.
Fast math where it's safe, a precise master copy where small numbers must survive.
Lower precision in training (16-bit numbers, kept dynamic) is not the same as quantization for deployment (often 8-bit or less, frozen). Training needs enough precision to learn small updates; a finished model can usually be squeezed far harder afterward.