JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Governing frontier risk

Zoom out from individual attacks to the policies that decide when a model is safe enough to release — and how the whole stack fits together.

From per-attack defense to release decisions

The first four guides looked through a microscope at individual attacks and defenses. Now pull back. As models grow more capable, the binding question stops being "did we patch this jailbreak?" and becomes "is this model, as a whole, safe enough to release to the world — and if not, what would have to be true first?" Answering that needs more than engineering; it needs policy: written, public commitments that connect measured capability to required safeguards.

Capability rises predictably as models scale — the trend that turns "did we patch this jailbreak?" into "is it safe to release at all?"

Log-log curve of loss falling steadily as model and data scale grow.

Frontier safety policies: if-then commitments

Leading labs now publish frontier model safety policies — documents under names like Responsible Scaling Policy, Preparedness Framework, or Frontier Safety Framework. Despite different branding they share a skeleton: define capability thresholds (red lines a model might cross), bind each threshold to required safeguards that must be in place before training or deploying past it, and commit to halt or restrict if the safeguards are not ready. It is an if-then structure: if a model demonstrates a dangerous capability above some level, then a specified set of security, deployment, and oversight measures becomes mandatory.

  1. Measure — run dangerous capability evaluations and red-teaming to locate the model against the thresholds.
  2. Match safeguards to capability — higher capability triggers stronger weight security, tighter deployment controls, and heavier monitoring.
  3. Gate the release — if the required safeguards are not met, the model does not ship at full capability, full stop.
  4. Revisit continuously — thresholds and evals get revised as understanding improves and as the frontier moves.
The if-then commitment as a lifecycle: measure capability, match safeguards, gate the release, then revisit continuously as thresholds and models change.

A cyclical pipeline with a monitoring-and-update feedback loop.

Why this is hard to get right

These policies are genuine progress, but be honest about their limits. Capability thresholds are judgment calls on fuzzy science — we cannot yet measure "meaningful uplift" precisely. The evals themselves can be gamed or simply miss a capability, and the dual-use nature of every skill means a clean line between safe and unsafe rarely exists. There is also a competitive pull: a lab that gates hard while rivals do not pays a cost, which is why many frameworks lean on external verification, third-party evals, and the prospect of regulation to keep commitments credible.

The whole stack, in one picture

Step back over the five guides and the layers line up. At the behavior layer, safety fine-tuning and refusal shape what the model wants to do. At the runtime layer, guardrails, moderation, and system prompts wrap it. At the attack layer, jailbreaks, injections, poisoning, and backdoors probe for gaps. At the assurance layer, red-teaming and capability evals measure what is left. And at the governance layer, frontier policies decide what may ship. No layer is sufficient; together they make LLM safety a moving, defense-in-depth practice rather than a solved problem.

P(\text{breach}) = \prod_{i=1}^{n} p_i

Defense in depth, quantified: an attack only wins if it slips through every layer, so the breach probability is the product of each layer's pass-through.