DoRA (weight-decomposed low-rank adaptation)

Think of every weight column as an arrow: it has a length (magnitude) and a heading (direction). Full fine-tuning is free to change both, and analysis shows it does so in a way LoRA cannot easily imitate — LoRA tends to move magnitude and direction together in lockstep. DoRA's idea is to split the arrow explicitly, learning the lengths with their own free parameters and steering only the headings with a LoRA-style low-rank update.

Mechanically, DoRA decomposes the pretrained weight into a magnitude vector m, one scalar per column, and a unit-norm direction matrix V over its columns. It then trains m directly and adds a low-rank correction to V before renormalizing each column back to unit length. This restores a degree of freedom LoRA collapses: the model can adjust how strongly a feature fires independently of which way it points, and empirically the learning pattern across training comes to resemble full fine-tuning much more closely.

The payoff is accuracy closer to full fine-tuning at the same low parameter budget, especially at small ranks where plain LoRA struggles. Costs are modest extra compute for the normalization and a slightly more involved merge, but like LoRA the trained result can be folded back into a single weight matrix for inference.