adversarial robustness

Adversarial robustness is a model's ability to keep giving correct answers even when its inputs are maliciously perturbed within a small budget. Where ordinary accuracy is measured on clean, naturally occurring images, robust accuracy is measured on worst-case images: for each test input, an attacker is allowed to perturb it within an ε-ball and the model is scored on whether it still answers correctly under that adversarial perturbation. A model with 95% clean accuracy might have only 40% robust accuracy under a strong attack — the gap quantifies its fragility.

Formally, the goal is robust risk minimization, often written as a min-max (saddle-point) problem: minimize over the model's weights the expected value of the maximum-over-perturbations of the loss, where the inner maximization ranges over all δ with ‖δ‖ ≤ ε. The inner 'max' is the strongest attack the threat model permits; the outer 'min' trains the model to do well against that worst case. This framing (Madry et al., 2018) unifies attack and defense: a defense is only as meaningful as the strongest attack it withstands, so robustness must always be stated relative to a specified threat model (the norm, ε, and attacker knowledge).

The leading practical defense is adversarial training: during training, generate adversarial examples on the fly (typically with PGD) and train the model to classify those, not just clean images. This directly approximates the min-max objective and is the most reliable empirical defense, though it is several times more expensive than normal training and can overfit to the specific attack used. A different branch, certified defenses (e.g. randomized smoothing, interval-bound propagation), provides a mathematical guarantee that no perturbation within a radius can change the prediction — weaker numbers but a proof rather than a hope, which empirical defenses cannot offer.

A central, well-documented phenomenon is the robustness–accuracy tradeoff: making a model more robust usually lowers its clean accuracy, because robust features are a smaller, more conservative subset than the brittle-but-predictive features standard training exploits. Robustness also tends to demand much more data and capacity. A recurring trap in the literature is the illusion of robustness from gradient masking or obfuscated gradients — defenses that merely hide the gradient defeat weak attacks while remaining broken under adaptive attacks, so trustworthy evaluation requires adaptive, attack-aware testing (the AutoAttack suite and similar) rather than a single fixed attack.