QLoRA (quantized low-rank adaptation)

Fine-tuning a large model normally means holding its billions of weights, their gradients, and an optimizer state in memory at once, which prices a 65B model out of any single consumer card. QLoRA's trick is to keep the frozen base model in a brutally compact 4-bit form and only train a thin pair of low-rank adapters on top. The heavy weights are read-only ballast; the tiny adapters are the only thing that moves, so the memory bill collapses to something one 48GB GPU can pay.

Three pieces make it work without wrecking accuracy. The base is stored in 4-bit NormalFloat, a data type whose quantization levels are spaced to match the roughly Gaussian distribution of trained weights, so each bucket carries equal probability mass. Double quantization then quantizes the quantization constants themselves to claw back a little more memory. During the forward and backward pass each 4-bit block is dequantized on the fly to bf16, the gradient flows through it into the LoRA matrices, and paged optimizers spill optimizer state to CPU to survive memory spikes.

The result preserves nearly full-precision fine-tuning quality while running on hardware a hobbyist owns. The caveat is that the base stays quantized: QLoRA adapts a 4-bit model, so any precision the quantization already destroyed is gone, and merging the adapter back into a 4-bit base is lossy unless you re-dequantize to higher precision first.