Fine-Tuning & Adaptation

LoRA (low-rank adaptation)

LoRA is the most popular way to fine-tune cheaply. The insight is that the update a model needs for a new task is simple in a precise sense — it can be captured by a couple of small, skinny matrices instead of a huge dense one. So you freeze the original weight and learn only that compact update.

For a weight matrix W, LoRA learns two small matrices A and B and represents the change as their product BA, which has low rank. During training only A and B are updated; W stays frozen. The two matrices together hold a tiny fraction of W's parameters, so you train and store far less. At inference you can either fold BA back into W at no extra cost, or keep it separate so you can hot-swap tasks.

Because each LoRA is just a few megabytes, you can ship many of them per base model and load the right one on demand. It is the workhorse of practical open-model customisation.

W' = W + BA,\quad B\in\mathbb{R}^{d\times r},\ A\in\mathbb{R}^{r\times k},\ r\ll d,k

Only the low-rank product BA is trained; the rank r is small (often 8 to 64), so A and B hold far fewer numbers than W.

Also called

low-rank adaptation