Parameter-Efficient Fine-Tuning: The LoRA Family

The big idea: adapt, don't overwrite

PEFT rests on an empirical observation: the change a fine-tuning task makes to a model's weights is small and structured, even though the model itself is enormous. So instead of editing the billions of pretrained weights, you freeze them and add a small, trainable delta on the side. At inference you compute `W·x + delta·x`. You train only the delta, store only the delta, and can swap deltas to switch tasks instantly.

This reframes adaptation from editing a brain to attaching a small accessory. The base model's general knowledge is preserved untouched (less catastrophic forgetting), and each task costs megabytes rather than a full model copy. The whole family below is variations on what the accessory looks like.

LoRA: a low-rank update

LoRA (Low-Rank Adaptation) is the workhorse. The trainable delta for a weight matrix W of shape d×d is not a full d×d matrix — that would be as big as W. Instead LoRA factors it as the product of two thin matrices, B (d×r) and A (r×d), with rank r tiny (often 8–64). The number of trainable parameters drops from d² to 2dr. For a 4096×4096 layer at r=16, that is ~131k parameters instead of ~16.8M — a hundredfold cut.

W' = W + \frac{\alpha}{r}\,B A,\qquad B\in\mathbb{R}^{d\times r},\ A\in\mathbb{R}^{r\times d},\ r\ll d

LoRA replaces the full d×d weight update with a low-rank product BA, scaled by alpha/r.

# LoRA: W is frozen; only A, B are trained
# A ~ N(0, sigma), B = 0  ->  delta starts at zero (no-op at init)
h = x @ W.T                      # frozen base path
h = h + (x @ A.T @ B.T) * (alpha / r)   # low-rank adapter path

B is initialized to zero so the adapter is a no-op at the start of training; alpha/r scales the update.

Two knobs matter: rank r (capacity of the adapter) and alpha (a scaling factor; the effective scale is alpha/r). After training you can merge the adapter back into W by computing `W + (alpha/r)·B·A` once — then inference has zero extra cost and zero extra latency, because the adapted model looks exactly like an ordinary one. Keep adapters separate if you want to hot-swap tasks; merge them if you want a single frozen deployment artifact.

QLoRA: fine-tune in 4 bits

LoRA shrinks the gradient and optimizer memory, but you still hold the full base weights. QLoRA attacks that last cost: it stores the frozen base model in 4-bit quantization (a custom NF4 format tuned for normally-distributed weights) and trains LoRA adapters on top in higher precision. The forward pass dequantizes weights block-by-block on the fly; gradients flow only into the small adapters, which stay in 16-bit. Result: you can fine-tune a 65B model on a single 48 GB GPU.

Y = X\,\mathrm{dequant}_{\text{NF4}}(W) \;+\; X\,\frac{\alpha}{r}\,B A

QLoRA keeps the frozen base weight in 4-bit NF4 and dequantizes it on the fly, then adds the trainable LoRA term.

Beyond LoRA: DoRA and the prompt-space methods

DoRA (Weight-Decomposed Low-Rank Adaptation) notices that a weight update changes two things at once: a vector's magnitude and its direction. DoRA decomposes each weight column into a magnitude scalar and a unit-direction vector, then applies LoRA only to the direction while training the magnitude separately. This decoupling lets the adapter behave more like full fine-tuning, closing much of the small quality gap LoRA leaves — at almost no extra inference cost once merged.

A different branch leaves the weights entirely alone and instead learns inputs. Soft-prompt tuning prepends a handful of trainable embedding vectors — a soft prompt — to the input sequence; the model is frozen, and only those continuous vectors are learned. It is the most parameter-thrifty method of all (a few thousand numbers), though it usually needs a large base model to work well and tends to trail LoRA in quality.

Prefix tuning is the more powerful cousin: instead of only prepending to the input layer, it prepends trainable key/value vectors at every attention layer. That gives the soft prompt far more leverage over the model's internal computation, recovering much of LoRA's quality while still touching no original weight. Think of soft prompts as steering from the door and prefix tuning as steering from inside every room.

Choosing a method in practice

Default to LoRA or QLoRA. They are the best-understood, best-tooled, and most reliable. Use QLoRA when GPU memory is the binding constraint; plain LoRA when you have headroom and want a touch more quality.
Reach for DoRA when LoRA's quality gap matters for your task and you can afford slightly more training compute; it is close to a drop-in upgrade.
Tune the rank, not just the learning rate. Too-low rank under-fits; too-high rank wastes parameters and can overfit. Sweep r over {8, 16, 32, 64} and watch held-out loss.
Apply adapters to all linear layers, not just attention's query/value — adapting the MLP and projection matrices too usually helps more than raising rank.
Reserve prompt/prefix tuning for when you must keep one frozen base serving many tasks with truly minimal per-task storage, and you can tolerate a quality trade-off.