prefix tuning

Prompting steers a frozen model by putting words in front of the input; prefix tuning does the same but with vectors the model would never have written, optimized directly. Instead of prepending tokens at the input only, it prepends a short block of trainable continuous vectors to the keys and values of attention at every layer. The frozen transformer then attends to this learned context as if it were earlier tokens, and gradient descent shapes those vectors to elicit the behavior you want.

Because the prefix lives in the key-value cache of each layer, it influences computation all the way up the stack, not just at the embedding layer — this is what separates it from input-only soft prompts and gives it more leverage. Training only the prefix (often reparameterized through a small MLP to stabilize optimization, then discarded) leaves all original weights frozen, so the cost is a few thousand vectors per task. At inference the prefix is simply concatenated into the KV cache.

Prefix tuning shines when you need many lightweight task specializations over one shared backbone and want them to act deeply rather than just at the prompt. Its weakness is expressivity relative to LoRA or full fine-tuning, sensitivity to prefix length, and the awkwardness of reasoning about what those opaque continuous vectors actually encode.