JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Red-teaming: breaking your own model on purpose

The discipline of adversarial evaluation — finding the failures before release, measuring dangerous capabilities, and reasoning about dual use.

Why you attack your own model

Red-teaming borrows its name from military and security practice: a dedicated team plays the adversary and tries to make the system fail, so the failures surface in a lab instead of in the wild. For an LLM, that means deliberately eliciting the harmful, biased, or policy-violating outputs the model is supposed to refuse — then feeding those failures back into safety fine-tuning and guardrails. Red-teaming is not a one-time gate before launch; it is a continuous loop, because every capability upgrade opens new failure surface.

x^{\star} = \arg\max_{x \in \mathcal{X}} \; P_{\theta}(\text{harmful} \mid x)

Red-teaming as search: the adversary hunts for the input that maximizes the chance of a harmful output.

Manual and automated red-teaming

Human red-teamers bring creativity and domain expertise — a real biologist or a real social engineer probes failure modes a generic tester never imagines. But humans are slow and expensive, so the practice now pairs them with automated jailbreak search: attacker models that generate thousands of adversarial prompts, LLM judges that score the responses, and the optimization techniques from the previous guide turned defensive. The output is coverage at scale; the humans then concentrate on the hardest, highest-stakes categories.

  1. Define the threat model — who is the attacker, what are they after, what access do they have? Vague goals produce vague tests.
  2. Elicit failures — manual probes plus automated search across each harm category.
  3. Triage and reproduce — score severity, confirm the failure is reliable, not a one-off fluke.
  4. Mitigate and re-test — patch via fine-tuning or filters, then attack again to check the fix did not just move the bump under the rug.
Red-teaming is a feedback loop: define the threat model, elicit failures, triage and reproduce them, mitigate via fine-tuning, then re-test — and repeat.

Diagram: a cyclical lifecycle pipeline with a feedback loop that returns to earlier stages.

Dangerous capability evaluations

Most red-teaming asks "will the model say something bad?" A deeper class of testing asks "can the model do something bad?" Dangerous capability evaluations measure whether a model meaningfully uplifts a malicious actor in high-consequence domains: does it provide a non-trivial advantage in synthesizing a pathogen, writing novel malware, running a persuasion or influence operation, or acting autonomously to acquire resources and evade oversight? The bar is not "can a determined expert already find this on the internet" but "does the model lower the barrier for someone who otherwise could not."

Dangerous-capability evals often run the model as an agent — reason, act, observe — to test what it can actually do, not just what it says.

Diagram: an LLM agent loop cycling through reason, act, and observe steps.

The dual-use bind

Almost every dangerous capability is the same capability that makes the model valuable. The reasoning that helps a chemist also helps a poisoner; the coding skill that ships features also writes exploits; the persuasive fluency that drafts a great cover letter also drafts disinformation. This is the dual-use problem, and it means you can rarely just "remove" a danger — removing it usually removes the usefulness too. Safety work therefore aims at the misuse path (refusing the operational request, watermarking, monitoring) rather than lobotomizing the underlying knowledge, all while watching the safety-helpfulness trade-off from guide one.