a safety evaluation
Knowing a car is fast is different from knowing it is safe — for safety you test the brakes, the airbags, and what happens in a crash. The same split applies to AI. A capability evaluation asks how skilled a model is; a safety evaluation asks whether it behaves acceptably, especially under pressure: does it refuse harmful requests, avoid deception, stay honest, resist manipulation, and act within its intended limits?
A safety (or alignment) evaluation puts the model into situations designed to reveal bad behavior and checks how it responds. Examples: feeding it requests for dangerous instructions to see if it refuses; testing whether it lies when lying would help it; probing whether it flatters the user instead of telling the truth (sycophancy); or placing it in a simulated scenario to see whether it tries to resist being shut down. The output is evidence about behavior — 'refused 98% of harmful prompts in this test set' — rather than about raw skill.
The crucial honesty here is that passing a safety evaluation is evidence of safety on the tested cases, not a guarantee of safety in general. A test can only cover situations someone thought to include; real users and adversaries will find inputs the test missed (distribution shift), and a capable model might even behave well precisely because it senses it is being tested. 'Passed the eval' and 'is safe' are not the same claim — bridging that gap is the job of red-teaming and a structured safety case.
A safety eval feeds a model 1,000 harmful requests and reports it refused 992. That is reassuring for those 1,000 cases, but says little about the 1,001st request a determined attacker invents tomorrow.
A clean safety score covers the cases tested, not every case that will ever occur.
Commonly confused with capability evaluation: capability asks 'can it?', safety asks 'will it behave?'. And passing a safety eval shows safety on the tested distribution only — it is evidence, never proof, of safety.