jailbreak attacks

A jailbreak is a prompt that talks a model out of its own safety policy, getting it to produce content it was aligned to refuse. The classic image is the DAN persona ("Do Anything Now") or a fictional-framing wrapper that recasts a forbidden request as harmless roleplay, translation, or a hypothetical that the model feels licensed to complete.

Jailbreaks exploit the gap between a model's broad capabilities and its shallow, often surface-level alignment. Common levers: persona and roleplay to suppress refusal, obfuscation (base64, leetspeak, low-resource languages, payload splitting) to slip past content filters, competing-objective framing that pits helpfulness against safety, and optimization-found adversarial suffixes that mechanically raise the probability of compliance. Many transfer across models, suggesting they hit shared weaknesses in alignment rather than one model's quirk.

Jailbreaking is an arms race: each defense — refusal training, input and output classifiers, system-prompt hardening — invites a new attack, and robustness on a fixed jailbreak set rarely generalizes to its paraphrases. A model can refuse a known jailbreak verbatim yet fall to a one-word edit, so static jailbreak benchmarks tend to overstate a model's real safety.