LoRA (low-rank adaptation)

When you adapt a pretrained model to a new task, the change you actually need is usually small and structured — you are nudging a competent model, not rebuilding it. LoRA bets that this nudge, the weight update for a layer, lives in a low-dimensional subspace. So instead of training the full update matrix, it freezes the original weights and learns the update as the product of two skinny matrices, a down-projection and an up-projection, whose rank you pick to be far smaller than the layer width.

Concretely, for a frozen weight W0 the adapted layer computes W0·x plus a scaled correction (alpha over r) times B·A·x, where A maps the input down to rank r and B maps it back up. Only A and B train; A is typically random-Gaussian and B starts at zero so the model begins exactly as the base. Trainable parameters drop by orders of magnitude, optimizer memory shrinks with them, and at inference you can fold B·A back into W0 so there is no added latency at all.

Because adapters are tiny and the base is untouched, one frozen model can host many swappable LoRAs — one per task, tenant, or style — and a server can batch requests across them. The cost is expressivity: a rank that is too low cannot capture updates that genuinely need full rank, and the right rank, scaling, and target modules remain empirical choices.