diffusion inpainting

Diffusion inpainting is the task of filling in a masked-out region of an image so the result looks seamless — removing an object, replacing a sky, extending a painting, or painting in something new described by a prompt. The defining challenge is consistency: the filled pixels must agree with the surrounding image in lighting, perspective, texture, and content, and also satisfy any prompt, so the patch is genuinely indistinguishable from an original. A diffusion model is well suited to this because its iterative denoising can be steered to respect known pixels at every step.

There are two main ways to do it. The training-free approach (such as RePaint) takes a standard diffusion model and, at every reverse step, overwrites the known (unmasked) region of the current noisy image with the correctly-noised version of the real pixels, while letting the model freely denoise only the masked region. Because the known region keeps getting re-imposed, the model is forced to harmonize its fill with fixed surroundings; resampling tricks reduce seams at the mask boundary. The fine-tuned approach modifies the network to accept the mask and the masked image as extra input channels and trains specifically on inpainting; dedicated inpainting checkpoints (such as Stable Diffusion's inpainting model) do this and generally give cleaner, more controllable results.

Inpainting underpins a huge share of real-world creative editing: object removal, generative fill, garment and product swaps, restoration of damaged photos, and 'outpainting' (extending an image beyond its borders, which is just inpainting where the mask is everything outside the original). Its characteristic failure modes are boundary seams, content that ignores the surroundings (a shadow with no caster), and prompt leakage where the new content's style clashes with the photo. Done well, the seam is invisible; done poorly, the eye snaps straight to the patch.