full vs. PEFT fine-tuning

When you adapt a model you face a fork in the road: update every weight (full fine-tuning) or update a tiny, efficient slice (PEFT). Full fine-tuning is the heavyweight option — maximum flexibility, maximum cost; PEFT is the nimble option — most of the benefit for a fraction of the price.

Full fine-tuning needs memory for all parameters plus their gradients and optimizer state — often several times the model's own size — and yields one full-size checkpoint per task. PEFT freezes the base and trains under a percent of the weights, so memory and storage collapse and tasks become swappable adapters. For most adaptation work the quality gap is small; full fine-tuning still wins when you must move the model substantially — deep domain shifts, new languages, or large changes in behaviour.

The practical rule of thumb: start with PEFT, and escalate to full fine-tuning only when measurements show the gain is worth the extra cost.