When Pixels Lie: Adversarial Examples and How to Fight Back

The unsettling discovery

Here is a result that stunned the deep-learning community when it first appeared. Take a photo of a panda. Feed it to a strong image classifier, and it answers "panda" with 99% confidence — exactly what we want. Now add a faint layer of carefully chosen noise, so weak that to your eyes the picture is unchanged: same fur, same bamboo, same black eye-patches. Feed this new image to the same model. It now says "gibbon" — with 99.3% confidence. Not "I'm not sure." Confidently, catastrophically wrong. This is the famous panda-to-gibbon story, and it is not a glitch in one model on one image. It is a property of almost every neural network we know how to train.

Let's name the thing. An adversarial example is an input that a human reads one way but a model reads completely differently, because someone deliberately added a small, structured perturbation designed to fool the model. The word "deliberately" is doing a lot of work. This is not the random noise a camera adds in dim light, the kind any decent model shrugs off. It is a tailored nudge, computed against this specific model, pointed in exactly the direction that does the most damage for the least visible change.

Why should you care beyond the curiosity? Because vision models increasingly make decisions that matter. Researchers have stuck a few innocuous-looking stickers on a stop sign and made a self-driving car's detector read it as a speed-limit sign. Attackers have nudged the pixels of banned imagery just enough to slip past a content-moderation filter while looking unchanged to a human moderator. A face-recognition gate, a medical-imaging triage system, a fraud-screening scanner — anywhere a model gates a real-world action, the gap between average accuracy and worst-case robustness becomes a security hole. This guide is about understanding that hole and learning to close it.

Why do adversarial examples exist?

It's tempting to think adversarial examples mean our models are just badly trained, and that a bigger or smarter network would not fall for them. The truth is stranger and more fundamental, so let's build the intuition carefully rather than wave at it. The two ideas you need are high dimensionality and the decision boundary.

Start with dimensionality. A modest colour image of 224x224 pixels has 224 x 224 x 3 ≈ 150,000 numbers. Each is one dimension, so an image is a single point in a space with about 150,000 axes. Now here is the key arithmetic of high dimensions: if we change every one of those 150,000 numbers by a tiny amount — say a brightness step so small no eye could spot it on any single pixel — the changes do not stay tiny when we add them up. A whisper per pixel, multiplied across 150,000 pixels and aligned in just the right direction, becomes a shout. The attacker spends an invisible budget on each pixel but collects a large total push, because there are so many pixels to push.

Now the decision boundary. A trained classifier carves its high-dimensional input space into regions, one per class, separated by boundaries — surfaces where the predicted label flips from, say, "panda" to "gibbon." The unsettling empirical fact is that real images tend to sit surprisingly close to these boundaries. Locally, near any input, a deep network behaves almost like a straight ruler (it is roughly linear in a small neighbourhood), so to cross a boundary you don't need to travel far — you just need to head straight at it. Combine this with the high-dimensional leverage above: a tiny coordinated step per pixel, aimed perpendicular to the nearest boundary, is enough to step over it into the wrong region.

The last piece is why humans don't fall for this but networks do. Your visual system latches onto robust, semantic structure — the shape of the body, the proportions of the face, the layout of limbs. A convolutional network, trained only to minimise classification error, is free to also exploit non-robust features: faint, high-frequency statistical patterns in the pixels that happen to correlate with the right label on the training data but carry no meaning a human would recognise. Those features are genuinely predictive on clean data, which is why the model learns them — but they are brittle, and they are exactly what an adversarial perturbation rewrites. The attacker isn't adding "gibbon" texture you could see; it is quietly editing invisible features the model secretly relies on.

Building an attack: the Fast Gradient Sign Method

Now we build our first actual attack, slowly, because it is the template every later attack riffs on. First recall how training works, because the attack is its mirror image. To train, we compute the loss J — a number measuring how wrong the model's prediction is — and then the gradient of that loss with respect to the WEIGHTS. The gradient points in the direction that would increase the loss fastest, so training takes a small step in the OPPOSITE direction, gently adjusting the weights to make the loss smaller. Repeat millions of times and the model gets good. The image stays fixed; the weights move.

The Fast Gradient Sign Method — FGSM — performs one elegant inversion. Freeze the weights (the model is already trained and we don't get to change it). Instead, take the gradient of the loss with respect to the INPUT PIXELS. That gradient answers a new question: which way should I change each pixel to make the loss BIGGER — that is, to make the model more wrong? Then, instead of stepping downhill like training, we step UPHILL on the loss, pushing the image toward greater error. One step that direction and we may have an adversarial example.

FGSM: take the sign of the input-gradient of the loss, scale it by a small budget, and add it to the clean image.

Let's decode every symbol, because this one formula is the whole guide in miniature. x is the clean input image (the real panda). θ (theta) is the model's weights, and they are FROZEN here — we never change them, the bracket just reminds us the loss depends on them. y is the true label ("panda"). J(θ, x, y) is the loss: how wrong the model is on this image given the true label. ∇_x J (read "the gradient of J with respect to x") is a number for every single pixel, telling us how much nudging that pixel up or down would raise the loss — i.e. the per-pixel direction that makes the model more wrong. sign(·) throws away the magnitude of each of those numbers and keeps only its direction: +1 if the gradient was positive, −1 if negative. Finally ε (epsilon) is the perturbation budget — a small fixed number, like 0.01 on a 0-to-1 pixel scale — that sets how big a step we take.

Two questions deserve a clear answer. First, why take the SIGN instead of using the raw gradient? Because the raw gradient is huge on some pixels and tiny on others, so following it would dump most of the change onto a few pixels — visible blotches. Taking the sign spends an EQUAL tiny budget of ε on every pixel: each pixel moves by exactly +ε or −ε, never more. That is what keeps the change invisible while still pushing every pixel the helpful way. This is also why we say FGSM bounds the change in the L-infinity sense — no single pixel ever moves by more than ε (we'll formalise that norm next section). Second, why move UP the loss? Training moves down to make the model right; we move up to make it wrong. Up the loss gradient is, by definition, the locally fastest route to higher error — straight at the nearest decision boundary.

Notice the tradeoff ε controls. Make ε smaller and the attack is more invisible but weaker — it may not push the image all the way across the boundary. Make ε larger and the attack is stronger but starts to leave faint visible artefacts. The whole game of adversarial robustness lives inside this dial: how much can the model survive within a budget so small a human can't even see it was used?

import torch
import torch.nn.functional as F

def fgsm_attack(model, x, y, epsilon):
    # x: a clean image batch, y: the true labels, model: a trained (frozen) classifier
    x = x.clone().detach()
    x.requires_grad_(True)          # we want the gradient w.r.t. the INPUT, not the weights

    # 1. Forward pass: how wrong is the model on this image?
    logits = model(x)
    loss = F.cross_entropy(logits, y)

    # 2. Backprop the loss all the way back to the input pixels
    model.zero_grad()
    loss.backward()                 # fills x.grad with the per-pixel gradient of the loss

    # 3. Keep only the DIRECTION of each pixel's gradient (+1 / -1)
    perturbation = epsilon * x.grad.sign()

    # 4. Step UPHILL on the loss to make the model more wrong
    x_adv = x + perturbation
    x_adv = torch.clamp(x_adv, 0.0, 1.0)   # keep pixels in a valid image range
    return x_adv.detach()

Threat models and stronger attacks

Before we build stronger attacks, we have to be precise about the rules of the game. In security this is called the threat model, and it answers two questions: what can the attacker SEE, and what can the attacker CHANGE? Get sloppy here and you will fool yourself into thinking a defense works when you simply tested it against a weak attacker.

On what the attacker can SEE, there are two classic settings. In a white-box attack the attacker knows everything: the architecture and the exact weights θ, so they can compute ∇_x J directly — this is the strongest, most honest setting to test against. In a black-box attack the attacker cannot see the weights and can only poke the model with inputs and read its outputs. Black-box sounds safer for the defender, but it is more realistic for real deployments, and as we'll see it is more dangerous than it looks.

On what the attacker can CHANGE, we cap the perturbation with a norm. The constraint is written as the budget on how far x_adv may stray from x.

The allowed perturbations live inside an L_p ball of radius ε around the clean image.

Read this as: the difference between the adversarial image and the clean image, measured by the L_p norm, must not exceed ε. The subscript p chooses HOW we measure size, and it matters a lot. With p = ∞ (L-infinity), the norm is the single largest change among all pixels, so the constraint says "no individual pixel may move by more than ε" — that is exactly the budget FGSM respects when it adds ±ε everywhere. With p = 2 (L2, ordinary Euclidean distance), the norm measures the total energy of the change across all pixels, so the constraint says "the overall amount of change, summed up, must stay small" — this allows a few pixels to move a bit more as long as the total stays within budget. The set of all images obeying the constraint is a ball of radius ε around x; the attacker may roam anywhere inside it but not step outside, which is what keeps the perturbation invisible.

Now the upgrade from FGSM. Projected Gradient Descent (PGD) is, in one sentence, iterated FGSM with a leash. Instead of one big sign-step, it takes many small sign-steps, and after each step it projects the image back inside the L_p ball so the total budget is never exceeded. Because it re-computes the gradient after every step, it can curve toward the boundary as the landscape bends, finding a far worse perturbation than a single straight shot.

PGD: an FGSM-style step of size α, then snap back inside the ε-budget. Repeat for many iterations.

Decoding it: x_t is the perturbed image at step t (we start at x_0 = x, the clean image, sometimes plus a tiny random jitter). At each step we take an FGSM-style move, but with a smaller step size α (alpha) instead of the full ε, because we will be taking many steps and don't want any one of them to overshoot. ∇_x J(θ, x_t, y) is the input-gradient evaluated at the CURRENT perturbed image x_t — that re-aiming is what makes PGD stronger than FGSM. Finally Clip_{x,ε}(·) is the projection: after the step, any pixel that wandered more than ε from its original value in x is snapped back to the edge of the budget. The picture: PGD walks repeatedly toward higher loss but is leashed to the ε-ball, so it searches harder while staying just as invisible. This is why PGD is the standard strong attack people use to STRESS-TEST a defense — if your model only survives FGSM but folds under PGD, it was never really robust.

Defending: adversarial robustness

Time to defend. First we have to change what we even mean by "good." Ordinary accuracy asks: on average, over typical inputs, how often is the model right? Adversarial robustness asks a tougher question: for each input, if an adversary is allowed to perturb it anywhere within the ε-budget, how often does the model STILL get it right? This is worst-case accuracy, not average-case. A model can have 95% clean accuracy and 2% robust accuracy — meaning an attacker can break it on 98% of images. Reporting only the first number while shipping into an adversarial setting is the exact failure this whole track keeps warning about.

The leading defense is beautifully direct: if the model fails on adversarial examples, then train it on adversarial examples. This is adversarial training. During each training step, before updating the weights, you first run an attack (typically PGD) to generate the worst perturbation of the current batch, and then you teach the model to classify THOSE corrupted images correctly. The model is, in effect, vaccinated — repeatedly exposed to the strongest attacks it will face so it learns features that survive them. We can write this as a single min-max objective.

Robust optimisation: an inner adversary maximises the loss within the budget; an outer trainer minimises that worst case.

Read this from the inside out — it is a battle between two players. The inner part, max over δ (delta) with ||δ||_p ≤ ε, is the ATTACKER: given the current weights, find the perturbation δ inside the budget that makes the loss J(θ, x + δ, y) as LARGE as possible — the worst case for this one example. (In practice we approximate this max by running PGD.) Wrapping that, E_{(x,y)} is the EXPECTATION, an average over all the (image, label) pairs in our data — so we care about the worst case averaged across the whole dataset, not just one picture. The outer part, min over θ, is the DEFENDER: choose the weights θ that make even that worst-case, averaged loss as SMALL as possible. In plain words: prepare for the worst, not the average. Train the model so that no small in-budget perturbation can push it into a mistake.

It helps to see adversarial training as the militant cousin of ordinary data augmentation. Standard augmentation shows the model random crops, flips, colour jitters and brightness changes, so it learns to ignore irrelevant variation. Adversarial training does the same thing but with a hostile, optimised variation: instead of a random change it injects the single most damaging change an attacker could make within ε. Same idea — broaden what the model has seen — but aimed at the worst case rather than typical noise.

Evaluating robustness honestly

We close with a maturity lesson that rhymes with Guide 2's warning about gaming a metric. In the arms race of defenses, dozens of published methods have claimed strong robustness and then been broken within weeks. The usual culprit has a name: gradient masking, also called obfuscated gradients. The defense doesn't actually make the model robust — it merely makes the loss gradient ∇_x J useless to the attacker: noisy, shattered, or zero. A gradient-based attack like FGSM or PGD then can't find a direction and reports near-zero attack success, which looks like fantastic adversarial robustness. It is a mirage.

Why is it a mirage? Because masking the gradient hides the boundary from THIS attacker, not from ALL attackers. The decision boundary is still right next to the image; the defense only removed the signpost pointing at it. Bring a stronger attack — one that estimates the gradient numerically, or a black-box attack that needs no gradient at all, or a transferred adversarial example crafted on a substitute model — and the supposed robustness evaporates. The robust accuracy that looked like 90% turns out to be 3%.

Step back and notice the boundary of everything we did here. An adversarial perturbation is a WORST-CASE input change: an intelligent adversary, optimising hard, within a tiny budget, to break you on purpose. That is the right model for a security threat. But most real-world failures have no adversary at all. The camera changes, the lighting shifts, the hospital uses a different scanner, the deployment city has weather your training set never saw. The input distribution simply drifts away from training — nobody engineered it, yet the model degrades anyway.

That is exactly where the next guide goes. We move from the adversarial worst case to the natural worst case: distribution shift — what happens when the world the model meets is no longer the world it was trained on, how to detect inputs that are out-of-distribution, and how to adapt across domains. Same humbling theme, no villain required.