adversarial attack and defense
/ ad-ver-SAIR-ee-ul uh-TAK and dee-FENSS /
An adversarial attack is the deliberate craft of inputs designed to make a model fail; a defense is any technique meant to stop it. It is a security arms race, much like the endless duel between virus writers and antivirus makers. The attacker probes for the inputs where the model breaks; the defender tries to patch those weak spots; the attacker finds new ones. Decades into this game, no defense has been found that holds up against a determined, well-informed opponent.
Attacks come in flavors. In a white-box attack the adversary knows the model's internals and can compute the exact perturbation to fool it; in a black-box attack they only see the model's outputs and must probe from the outside, yet this often suffices. Attacks can target what the model outputs at test time (evasion) or poison the training data so the model learns a hidden flaw from the start. The best-known defense, adversarial training, simply throws attacks into the training set so the model learns to resist them — at the cost of more computation and often some ordinary accuracy.
Why care? Because models increasingly guard things that matter: spam and fraud filters, content moderation, biometric locks, autonomous vehicles. Anywhere a motivated adversary benefits from a wrong answer, attacks are not hypothetical. The honest summary of the field is sobering: defenses raise the cost of an attack and block lazy ones, but provable, general robustness against a clever attacker remains an open research problem, not a product you can buy.
A team announces a new defense and reports 96% accuracy under attack. Months later, another team revisits it with an attack tuned to that specific defense — and accuracy collapses to near zero. This pattern, repeated across many papers, became a cautionary tale about evaluating defenses against only the attacks they were designed to stop.
Why broken defenses keep getting published — and why adaptive evaluation matters.
A defense must be tested against an adaptive attacker who knows the defense exists and adjusts. Many published defenses worked only because they accidentally obscured the model's gradients, fooling weak attacks; a stronger, adaptive attack walked right through them. Treat any "robust" claim with suspicion until it survives adaptive evaluation.