catastrophic risk

There is a meaningful difference between a tool that occasionally embarrasses its user and one that, in the wrong hands or on a bad day, could harm thousands of people. Catastrophic risk names the high end of that spectrum: the danger that an AI system contributes to harm on a very large scale, measured in many lives, severe damage, or major disruption to society.

More precisely, catastrophic risk usually refers to large-scale harms that fall short of ending humanity's future but are still grave, the tier just below existential risk. Analysts often group the pathways into three: misuse, where a person deliberately uses AI for harm (for example, an AI that meaningfully helps a non-expert design a pathogen or run a sweeping cyberattack); accidents, where a system behaves in unintended, unsafe ways even though no one meant harm; and structural risk, where AI quietly reshapes incentives and institutions in destabilizing ways (such as eroding trust through mass disinformation, or pushing rivals into a dangerous arms race). A single event need not be world-ending to count.

The category matters because it is the practical focus of much frontier governance: dangerous-capability evaluations, responsible scaling policies, and security requirements are largely aimed at catastrophic, not necessarily existential, harms. It is worth keeping the two distinct: a catastrophe is recoverable in principle, an existential catastrophe is not. Where exactly to draw the threshold (how many lives, how much damage) is a judgment call, and people weigh present, concrete misuse against speculative future scenarios very differently.