Where noise comes from
Zoom into almost any real photo — especially one shot indoors or at night — and the smooth surfaces are not smooth at all. They shimmer with a faint, restless grain. That grain is image noise: small random errors added to the true brightness of every pixel. It is not a mistake your camera made; it is a physical consequence of how light and electronics work. Before we can erase noise cleanly, we have to understand where it is born.
The biggest source in normal photography is photon shot noise. Light does not flow into a sensor like a smooth stream of water; it arrives as discrete particles — photons — that land at random moments. Imagine standing under a tin roof in light rain and counting raindrops in one second. Even if the average rate is exactly 100 drops per second, one second you might count 92, the next 108. The rate is steady, but the arrivals clump randomly. Each tiny sensor well counts photons the same way, so even a perfectly uniform gray wall produces slightly different counts in neighbouring pixels. That fluctuation is shot noise, and it is fundamental — no better camera can remove it, because it is in the light itself.
On top of shot noise sit three electronic contributions. Read noise is the small uncertainty added when the sensor's tiny accumulated charge is amplified and measured — like trying to read a faint number off a worn ruler. Thermal noise (dark current) comes from heat: warm silicon spontaneously kicks loose electrons that masquerade as light, which is why long exposures and hot sensors look grainier. Finally, quantization is the rounding error from squeezing a continuous voltage into a finite set of integer levels (for example 0–255). Each step throws away a sliver of information and adds a tiny error. Together these set a floor below which a single shot cannot be made cleaner.
Diagram of the imaging pipeline from scene light through lens and sensor to a digital pixel grid.
When we design denoisers, we do not model every physical detail. Instead we work with simplified noise models that capture the shape of the corruption. The two you will meet constantly are: additive Gaussian noise, the soft fuzzy grain where every pixel is nudged up or down by a small random amount (this is the everyday low-light shimmer); and salt-and-pepper (impulse) noise, where scattered pixels are flipped to pure white (255, 'salt') or pure black (0, 'pepper'), like specks of dust on the image. They look different and — crucially — they demand different cures, as we will see.
The additive noise model: observed = true + random nudge.
Read this equation as a recipe for one pixel. Here f(x,y) is the true, clean intensity at location (x,y) — the value a perfect camera would record. n(x,y) is a random noise value drawn fresh for that pixel; we often model it as Gaussian with mean 0 and variance \sigma^2 (written \mathcal{N}(0,\sigma^2)), where \sigma measures how violent the grain is. And g(x,y) is the observed noisy pixel you actually get. The message is simple: every measured pixel is the truth plus a random nudge. 'Zero-mean' is the key gift — the nudges are equally likely to be positive or negative, so if you could capture and average many independent shots of the exact same scene, the nudges would cancel and you would recover f. Concretely, suppose the true gray is f=120. In one shot n=+7 gives g=127; in another n=-5 gives g=115; in a third n=+2 gives g=122. Average them: (127+115+122)/3 = 121.3 \approx 120. The noise washes out. Denoising from a single image is the art of getting that averaging effect without a stack of photos to average.
The denoising goal and the fundamental tradeoff
Now we can state the job precisely. We are handed the noisy image g, and we want to recover an estimate of the clean image f that we never directly observed. That is image denoising: it is an estimation problem, a piece of educated guesswork. We can never be certain we got f exactly right, because the original truth was destroyed the moment the noise was added. The best we can do is produce a \hat{f} (read 'f-hat', meaning 'our estimate of f') that is plausibly close.
Here is the tension that haunts every denoiser. Noise lives in rapid, pixel-to-pixel changes — and so do the things we treasure: edges, fine texture, small text. If you smooth aggressively to kill the noise, you also blur away the very detail that makes the image worth keeping. If you smooth timidly to protect detail, the grain survives. Removing noise and preserving detail pull in opposite directions. Every filter in this guide is really a different answer to one question: how do I tell a noisy fluctuation apart from a real edge?
Think of restoring a dusty old painting. The dust is the noise; the brushwork is the signal. A lazy restorer wipes hard with a wet cloth — yes, the dust goes, but so does a layer of paint, leaving the masterpiece flat and smeared. A master restorer lifts the dust while leaving every brushstroke crisp. That selectivity — clean the flat regions, but do not scrub across the strokes — is exactly the magic the median and bilateral filters will give us later in this guide.
How do we know if a denoiser did well? The honest first tool is your own eyes: did the grain vanish while edges stayed sharp and no smeary halos appeared? For a number, the most common is PSNR (Peak Signal-to-Noise Ratio), measured in decibels (dB). It compares the denoised result against a known clean reference: higher dB means closer to the truth (roughly, 30 dB is okay, 40 dB is excellent). PSNR needs a ground-truth image, so it is used in experiments where we deliberately added noise to a clean picture and want to score how well we removed it. Keep this in your back pocket — as we meet each filter, we will judge it by both criteria: how much grain it removes, and how much detail it keeps.
Why averaging filters fail on outliers
In Guide 2 we met the mean filter and Gaussian blur. Both are linear filters: they slide a small window over the image and replace each centre pixel with a weighted sum of its neighbours. The mean filter uses equal weights (a plain average); the Gaussian uses bell-shaped weights that favour the centre. For soft Gaussian grain they work beautifully — averaging several noisy-but-similar neighbours lets the zero-mean nudges cancel, exactly the multi-shot trick from Section 2, except the 'shots' are now nearby pixels instead of separate photos.
But hand them salt-and-pepper noise and they fall apart. The reason is that an average has no defence against a single extreme value. One pixel stuck at 255 does not get cancelled by its neighbours — it gets spread over them. The mean does not remove the outlier; it dilutes it into a pale smudge that infects the whole window. Let us watch it happen with real numbers.
# A 3x3 window over a dark, smooth region.
# One pixel has been hit by SALT noise (value 255);
# the other eight are the true dark surface (around 20).
window = [[ 20, 22, 19],
[ 21, 255, 18], # <-- 255 is the impulse (an outlier)
[ 23, 20, 22]]
# MEAN filter: sum all nine values, divide by nine.
values = [20,22,19, 21,255,18, 23,20,22]
mean = sum(values) / 9 # = 420 / 9 = 46.7
# The true value here should be about 20.
# The mean filter reports ~47 -- more than DOUBLE the truth.
# Worse: the bright speck did not disappear. It got smeared
# into a gray blob covering the whole 3x3 neighbourhood.Step through it. The eight good pixels sit around 20, so the honest answer is 'about 20'. But the lone 255 adds a huge 255 to the sum of 420, and dividing by nine gives 46.7 — more than double the truth. And remember the window slides: as it moves, that same 255 will be inside the next window, and the next, so the single speck does not vanish, it replicates into a smeared gray blob roughly the size of the kernel. Gaussian blur softens the damage (the speck gets a smaller weight when off-centre) but cannot escape it — when the impulse sits dead centre it still gets the largest weight of all. Averaging is the wrong tool because it trusts every value equally, even a value that is obviously a lie.
The fix is to stop summing and start ranking. If instead of adding the nine values we sorted them and picked the one in the middle, that single outlier would be shoved to the far end of the sorted list and simply ignored. That idea — order, not sum — is the heart of the next section.
The median filter: order beats average
The median filter does exactly what we just hinted at: for each pixel, collect the values in its neighbourhood window, sort them, and output the middle value — the median. The median is the value with as many neighbours below it as above it. Because it is defined by position in a sorted list rather than by a sum, a couple of wild outliers cannot drag it: they just sit harmlessly at the ends of the list while the middle stays put.
Output each pixel as the median of the values inside its window.
Unpack the notation. \mathcal{W}(x,y) is the window (neighbourhood) centred on pixel (x,y) — for a 3×3 filter, the nine pixels around and including it. The set \{\, f(i,j) : (i,j) \in \mathcal{W}(x,y) \,\} just means 'all the pixel values f inside that window'. \operatorname{median}\{\dots\} says: sort that collection from smallest to largest and return whichever value lands in the exact middle. g(x,y) is the cleaned output pixel. There are no multiplications and no weights — that is the whole point. Compare this with the convolution of Guide 2, g = \sum_k w_k f_k, which is a fixed weighted sum: here there are no fixed weights w_k at all, so the median filter cannot be written as a convolution. We will return to that in a moment.
# Reuse the EXACT window from the previous section.
window = [[ 20, 22, 19],
[ 21, 255, 18], # the 255 salt speck is still here
[ 23, 20, 22]]
values = [20,22,19, 21,255,18, 23,20,22]
# MEDIAN filter: sort, then take the middle element.
values.sort() # -> [18, 19, 20, 20, 21, 22, 22, 23, 255]
# ^
# nine values, so the middle is index 4 (the 5th):
median = values[len(values)//2] # = 21
# Output is 21 -- right in the true dark range (~20).
# The 255 got sorted to the very END of the list and
# was thrown away. The speck is GONE, not smeared.Watch what happened. Sorted, the nine values are [18,19,20,20,21,22,22,23,255]. The poisonous 255 is now at the far right, the ninth slot — and the median only ever looks at the fifth slot, which is 21. The truth was about 20, and we got 21. The speck did not get diluted into a blob; it was annihilated, because the median quite literally does not care about extreme values, only about ranks. This is why the median filter is the standard cure for salt-and-pepper noise: it removes impulses outright instead of spreading them.
It also preserves edges remarkably well, and the reason is elegant. Picture a window straddling a sharp boundary — say five dark pixels (~20) on one side and four bright pixels (~200) on the other. A mean filter averages them into a muddy in-between gray (~100) that belongs to neither side, blurring the edge. The median, after sorting, lands on a value near 20 (the larger group wins the middle), so the output snaps cleanly to one side of the edge instead of inventing a blended value that was never really there. Edges stay crisp; only the impulses die.
The bilateral filter: blur that stops at edges
Now the centrepiece. The median is a champion against impulses, but for soft Gaussian grain we would still like the smoothing power of a Gaussian blur — without its fatal habit of bleeding across edges. The bilateral filter achieves exactly that. The idea is to take an ordinary Gaussian blur and give it a second brain: weight each neighbour by two factors at once — how close it is in space, AND how similar it is in intensity. A neighbour only gets a meaningful vote if it is both nearby and looks like the centre pixel.
Here is the human analogy. Imagine forming your opinion by averaging the views of people around you. A plain Gaussian blur asks only 'how physically close is this person?' — so if you are standing at the boundary between two very different crowds, you average across both and end up with a mushy compromise that represents nobody. The bilateral filter adds a second question: 'does this person clearly belong to a different group than me?' If yes, you ignore them, no matter how close they stand. You end up averaging only the people who are both near you AND like you — so your own group's view stays sharp and is never contaminated by the crowd across the line.
The bilateral filter: a weighted average where each weight is the product of a spatial and a range Gaussian.
Let us name every symbol carefully. p is the centre pixel we are computing a new value for, and q ranges over its neighbours in the window \mathcal{S}. f(q) is the neighbour's intensity (the value we might borrow), and f(p) is the centre's intensity. There are two Gaussian weighting functions. G_{\sigma_s}(\lVert p-q\rVert) is the spatial Gaussian: it depends on the geometric distance \lVert p-q\rVert between the two pixels, and \sigma_s — the 'reach' — sets how far the filter looks (bigger \sigma_s = wider blur). G_{\sigma_r}(\lvert f(p)-f(q)\rvert) is the range Gaussian: it depends on the intensity difference \lvert f(p)-f(q)\rvert, and \sigma_r — the 'similarity tolerance' — sets how different in brightness a neighbour may be before its vote is suppressed (bigger \sigma_r = more tolerant, closer to a plain blur). Finally W_p is the normaliser: the sum of all the weights, so that dividing by it makes the weights add to 1 and the output stays in the right brightness range (a true average, not an inflated sum).
The single most important detail is that the two Gaussians are multiplied. A product is large only when both factors are large: a neighbour must be BOTH near in space AND similar in intensity to earn a real weight. If either condition fails — too far away, or too different in brightness — the product collapses toward zero and that neighbour is effectively ignored. That 'AND' is precisely what protects edges, as the next worked example shows.
Walk it across a step edge. Suppose pixel p is on the dark side, f(p)=30, sitting right next to a bright wall at intensity 200. Consider two neighbours. Neighbour q_1 is one step to the left, also dark, f(q_1)=32: it is spatially close (spatial weight high) and intensity-similar, \lvert 30-32\rvert = 2 (range weight high) — product high, so it counts fully, and averaging it in cancels grain on the dark side. Neighbour q_2 is one step to the right, on the bright wall, f(q_2)=205: spatially it is just as close (spatial weight high), but the intensity gap is \lvert 30-205\rvert = 175 — far larger than a sensible \sigma_r of, say, 25, so the range Gaussian crushes its weight to nearly zero. q_2 is essentially ignored. The result: p is smoothed using only its fellow dark pixels, and the bright wall does not bleed in. The edge survives razor-sharp while the flat regions on each side get cleaned. That is edge-aware smoothing in one stroke.
import numpy as np
def bilateral_pixel(img, x, y, radius, sigma_s, sigma_r):
"""Compute one bilateral-filtered output pixel at (x, y)."""
center = img[y, x]
weighted_sum = 0.0 # numerator: sum of f(q)*weight
norm = 0.0 # denominator W_p: sum of weights
for dy in range(-radius, radius + 1):
for dx in range(-radius, radius + 1):
q = img[y + dy, x + dx] # neighbour intensity f(q)
# spatial Gaussian: depends on distance from center
spatial = np.exp(-(dx*dx + dy*dy) / (2 * sigma_s**2))
# range Gaussian: depends on intensity difference
rng = np.exp(-((center - q)**2) / (2 * sigma_r**2))
w = spatial * rng # the PRODUCT -- near AND similar
weighted_sum += w * q
norm += w
return weighted_sum / norm # divide by W_p -> true averageTwo consequences worth noting. First, the bilateral filter is content-dependent — its weights change at every pixel because they depend on the local intensities f(p),f(q). So, like the median, it is not a fixed convolution; you cannot precompute one kernel and slide it everywhere. Second, this is the conceptual ancestor of a whole family of edge-aware methods. Replace 'compare single intensities' with 'compare whole patches of pixels' and you get non-local means; push the idea further and you reach the guided filter and the learned, edge-respecting denoisers of modern deep denoising. Understand the bilateral filter and you hold the seed of all of them.
Choosing a denoiser, and what comes next
The single most important habit in denoising is to match the filter to the noise model — diagnose the corruption before you reach for a tool. Look closely at your noisy image first: scattered pure-black and pure-white specks scream salt-and-pepper; a uniform fuzzy shimmer over smooth areas is Gaussian grain. The right diagnosis points straight at the right cure.
- Salt-and-pepper / impulse specks → reach for the median filter. Order-based, so it deletes outliers outright while keeping edges crisp. A 3×3 median clears light speckle; go larger only if the corruption is dense.
- Mild Gaussian grain with edges and texture you want to keep → use the bilateral filter. If you need raw speed and a little edge-softening is acceptable, a plain Gaussian blur is the cheap fallback.
- Heavy noise where simple windows are not enough → step up to the modern frontier: non-local means, wavelet-based denoising, BM3D, or learned deep denoisers. More power, more compute.
A word on that frontier, named here only as a signpost. Non-local means generalises the bilateral idea: instead of comparing single intensities, it compares whole small patches and averages pixels whose surroundings look alike, exploiting the fact that natural images repeat themselves. Wavelet methods (which the final guide of this track will set up) denoise by transforming the image into frequency-and-scale components and shrinking the small ones that are mostly noise. And deep CNN denoisers push furthest of all.
Here is the bridge to the deep-learning tracks ahead. A learned CNN denoiser does not have its rules handwritten the way the median or bilateral filter does. Instead it is shown thousands of (clean, noisy) image pairs and learns from data which fluctuations are noise and which are real structure — discovering its own, far more flexible version of the bilateral filter's 'near AND similar' rule. The hand-built filters in this guide are the conceptual foundation those networks stand on; meet them now and the learned denoisers later will feel like a natural next step rather than magic.
One last thread to carry forward. Notice that everything in this guide tried to suppress rapid pixel-to-pixel change as noise — yet the sharpest, most useful changes in an image are its edges. The very fluctuations we fought here are, from another angle, the signal we most want to find. The next guide flips the goal: instead of smoothing changes away, we will measure them on purpose with gradients and the Sobel operator to find edges — and you will see why denoising and edge-finding are two sides of the same coin.