Red-Teaming and Adversarial Evaluation

What red-teaming means for LLMs

Benchmarks measure typical behaviour; red-teaming measures worst-case behaviour under an adversary. Borrowed from security, it means deliberately probing a model to surface harmful, unsafe, or policy-violating outputs before real users — or attackers — find them. The mindset is inverted from Guide 2: you are not sampling the average user, you are role-playing the cleverest and most motivated bad actor, and a single reproducible failure is a finding, not noise. This is a pillar of AI safety evaluation.

Two threat models you must not confuse

Jailbreaks target the model's own safety training: the attacker is the user, and a crafted prompt — role-play ('you are DAN'), hypothetical framing, obfuscation, or a long persuasive preamble — coaxes the model past its guardrails into producing content it was trained to refuse. Prompt injection is different and more insidious: the attacker is not the user but a third party who plants malicious instructions in content the model later reads — a web page, an email, a retrieved document — hijacking an LLM application into following the attacker's commands instead of the developer's.

Manual red-teaming

The first line is human creativity. A red team — domain experts, security researchers, and crucially people with diverse cultural and linguistic backgrounds — probes the model against a documented taxonomy of harms (e.g. weapons, self-harm, privacy, fraud, bias). Good practice is structured: define the threat model and policy, log every prompt and verbatim response, label each attempt as success or failure, and turn confirmed exploits into permanent regression tests so a future model cannot silently reopen them.

Manual red-teaming finds the deepest, most creative failures, but it is slow, expensive, and cannot cover the combinatorial space of attacks. It also carries a human cost — reviewers read disturbing content — which is itself a reason to automate the breadth and reserve humans for depth.

Automated red-teaming

Automated red-teaming scales the search by letting algorithms generate attacks. Three broad strategies dominate. Optimisation-based methods (e.g. GCG) treat the prompt as a variable and use gradients to find adversarial suffixes that maximise the probability of a harmful completion. Model-based methods use an attacker LLM, often itself trained with RL, to invent and refine jailbreaks against a target. Search-based methods mutate and recombine known attacks, using the target's responses as a fitness signal. The output is a far larger and more diverse attack set than humans alone could write.

\underset{s\in\mathcal{V}^{k}}{\min}\;\; -\log p_{\theta}\!\left(y^{\star}\mid x\oplus s\right)

Optimisation-based attacks like GCG search for an adversarial suffix s of k tokens that minimises the loss of a target affirmative response y⋆ given the request x.

Measuring safety honestly

The headline number is attack success rate (ASR): the fraction of adversarial attempts that elicit the forbidden behaviour, ideally graded by a classifier or an LLM judge and spot-checked by humans. But ASR alone is a trap. You must report it against a named attack set (ASR is meaningless without saying which attacks), watch the over-refusal rate on benign-but-sensitive prompts (a model that refuses everything has ASR 0 and is useless), and stratify by harm category. The right summary is a frontier, not a point: safety versus helpfulness, the same trade-off that Constitutional AI and refusal training try to push outward, and which jailbreak robustness research aims to measure.

\mathrm{ASR}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{1}\!\left[\,J\!\left(x_i,\,M(x_i)\right)=\text{harmful}\,\right]

Attack success rate: the fraction of N adversarial attempts whose model response M(xᵢ) a judge J grades as eliciting the forbidden behaviour.