jailbreaking LLMs

A jailbreak is a prompt crafted to talk a model out of its own safety training. The model was tuned to refuse certain requests, so the attacker reframes the request until the refusal no longer fires: pretend this is a movie script, you are a character with no rules, answer only in the voice of my late grandmother who used to read me napalm recipes, ignore all previous instructions. The harmful goal is unchanged; only the wrapping is different, and that wrapping is enough to flip the model from refusing to complying.

Jailbreaks work because safety fine-tuning teaches the model a pattern, not a law. It learns that requests phrased like typical harmful ones should be refused, but the space of phrasings is unbounded, and there are always reframings the training never covered. Common families include role-play and persona attacks, hypothetical or fictional framings, instruction-override prefixes, low-resource-language or encoding tricks, and many-shot prompts that flood the context with examples of the model happily complying.

The uncomfortable truth is that no deployed model is fully jailbreak-proof; researchers find new bypasses for every system. So jailbreaking is treated as an arms race measured in robustness, not a bug to be patched once: defenders track how much effort a working jailbreak now takes, and whether the worst thing a jailbroken model reveals is actually dangerous.