third-party evaluation

Would you trust a restaurant that inspected its own kitchen and gave itself a perfect grade? Probably not, which is why we send in outside health inspectors. Third-party evaluation is that same idea for AI: instead of a developer alone deciding whether its model is safe, an independent outside party tests it and reports what it finds.

In practice, a third-party evaluator (a government safety institute, an independent lab, or specialist auditors) is given some level of access to a model and runs its own safety and capability tests: probing for dangerous capabilities, red-teaming for jailbreaks, checking for deceptive or unsafe behavior. The 'third party' is what matters: not the developer (first party) and not the customer (second party), but a neutral outsider with no stake in the model passing. A worked example: a developer claims its model refuses to give bioweapon instructions; an external team, given access, finds a prompt that slips past the refusal and reports it, so the gap can be fixed before release.

Independent evaluation matters because self-assessment has a built-in conflict of interest, and because outsiders bring fresh adversarial creativity. But it is hard to do well. Evaluators often need deep access (weights, internals, or an unfiltered version) that companies are reluctant to grant for security and trade-secret reasons; without enough access they may only see a sanitized surface. And a clean report is evidence of safety on what was tested, not proof of safety everywhere, especially since a capable model could in principle hold back, or 'sandbag', during a test it knows it is taking.