safety fine-tuning

Safety fine-tuning is the training stage that teaches an already-capable model when to say no. A raw pretrained model will happily continue any text, including step-by-step harm; safety fine-tuning adjusts its weights so that, faced with requests in defined harmful categories, it produces a refusal instead. It uses the same machinery as the rest of alignment: supervised examples of good refusals, and preference data where human or AI judges prefer the safe response, folded into RLHF, constitutional methods, or direct preference optimization.

The aim is to make safe behavior the model's default tendency, not a fragile add-on, so that it generalizes to harmful requests no one wrote a rule for. That is also its central difficulty: the training covers a finite set of attacks, and the space of rephrasings is infinite, so safety fine-tuning reliably blocks the obvious cases while leaving room for novel jailbreaks. Teams therefore mine adversarial prompts (often with automated search) and fold the hardest examples back into the next round of training.

Two side effects matter. First, push too hard and the model over-refuses, declining benign requests — a real cost, sometimes called the alignment tax. Second, safety training is shallow relative to capability: the underlying dangerous knowledge is untouched, only the disposition to share it is changed, and a small amount of adversarial fine-tuning, even on benign-looking data, can strip the safety layer back off. Safety fine-tuning raises the cost of misuse; it does not make a model incapable of harm.