red-teaming
Before a bank trusts its own security, it hires people to try to break in — to think like an attacker and find the holes before a real criminal does. Borrowed from military and computer-security practice, red-teaming applied to AI means deliberately trying to make a model misbehave: searching for the prompts, tricks, and situations that get it to do things it is supposed to refuse.
A red team might try to coax a model into giving dangerous instructions, leaking private data, producing hateful content, or being manipulated into ignoring its rules. They use creative phrasing, role-play framings, hypothetical scenarios, encoded requests, and persistence — whatever finds a crack. The goal is not to prove the model is good but to find the worst things it will do, so those failures can be fixed (for example by more training) before release. The findings feed safety evaluations and the eventual safety case.
Red-teaming is powerful but inherently incomplete: it can show a model is vulnerable, but never that it is invulnerable — absence of a found attack is not proof no attack exists. It also depends on the imagination and effort of the red team, so it tends to miss attack types nobody thought of. Because manual red-teaming is slow and limited, it is increasingly paired with automated red-teaming to widen coverage. Used well, it is one of the most practical safety tools we have; mistaken for a guarantee, it is dangerous.
A red teamer fails to get harmful instructions by asking directly, then succeeds by wrapping the request in a fictional story 'for a novel', revealing a gap the safety training missed.
Red-teaming surfaces concrete failures by attacking like an adversary — but a clean day proves only that this team, this time, found nothing.
Red-teaming can demonstrate insecurity but never security: not finding an attack is not the same as there being none. Its coverage is limited by the testers' creativity, which is why it is increasingly automated and combined with other methods.