Parameter-Efficient Fine-Tuning: LoRA, QLoRA & Adapters

The cost of moving every weight

A full fine-tune updates all of a model's parameters. For a 7-billion-parameter model that means holding the weights, their gradients, and the optimizer's running statistics — often more than 80 GB of GPU memory, several times the model's own size. That puts full fine-tuning out of reach for most people and makes every experiment slow and expensive.

Parameter-efficient fine-tuning (PEFT) sidesteps the whole problem with one bet: you can capture most of what fine-tuning needs by freezing the giant base model and training only a small number of new parameters. Empirically, that bet pays off remarkably often.

\text{trainable params}:\quad d\,k \;\longrightarrow\; r\,(d+k),\qquad r \ll \min(d,k)

Full fine-tuning trains all d×k weights of a matrix; PEFT trains only r(d+k) — a tiny fraction when the rank r is small.

LoRA: the workhorse

LoRA (Low-Rank Adaptation) is the method most people mean when they say PEFT. The insight is that the change a fine-tune makes to a big weight matrix is usually low-rank — it can be well approximated by multiplying two skinny matrices together. So instead of editing the frozen weight, LoRA learns those two small matrices alongside it and adds their product back in. Typically under 1% of the parameters end up trainable.

W' = W_0 + \Delta W = W_0 + \frac{\alpha}{r}\,B A,\quad B\in\mathbb{R}^{d\times r},\; A\in\mathbb{R}^{r\times k},\; r\ll\min(d,k)

LoRA freezes the base matrix W₀ and learns its update as a low-rank product BA, scaled by α/r.

# A typical LoRA setup
rank  r        = 16                 # width of the low-rank update
lora_alpha     = 32                 # scaling applied to the update
target_modules = [q_proj, k_proj,   # which weight matrices get an
                  v_proj, o_proj]   #   adapter (often the attention ones)
dropout        = 0.05
# Base weights stay frozen; only the small r-sized matrices train.

The few knobs that matter: rank, alpha (scaling), and which matrices to target.

QLoRA: fine-tune a big model on one GPU

QLoRA pushes the memory savings further by combining LoRA with quantization: it loads the frozen base model in 4-bit precision, then trains LoRA adapters in normal precision on top. The base barely takes any memory because it's quantized and never updated, while the trainable adapters stay accurate. This is what lets people fine-tune a 13B–70B model on a single consumer or modest cloud GPU.

Adapters and soft prompts

LoRA is one branch of a larger family. Adapter modules insert small new bottleneck layers between the frozen transformer layers and train only those — an older idea that LoRA partly streamlined. Soft prompt tuning goes even lighter: it freezes the entire network and learns a handful of continuous embedding vectors that are prepended to the input, a kind of trainable prompt that lives in vector space rather than in words.

These trade a little accuracy for even fewer trainable parameters and tinier artifacts. In practice LoRA and QLoRA dominate today, but knowing the family helps you recognize the shared idea: freeze the giant, train something small.

Full vs PEFT: when each wins

Reach for PEFT (LoRA/QLoRA) by default: it's cheap, fast to iterate, easy to host many variants, and matches full fine-tuning on most behavior and style tasks.
Consider a full fine-tune when you're teaching a large new capability or absorbing a lot of new domain knowledge, and you have the hardware and data to justify it.
Whichever you pick, evaluate against the original model too — sometimes the base already does the job and the fine-tune only adds risk.