neural radiance field

A neural radiance field, or NeRF, is a radically different way to store a 3D scene: not as points or meshes, but as a continuous function hidden inside a small neural network. The network takes a location in space and a viewing direction, and outputs the color and the density of the scene at that spot. Query it everywhere along the rays of a virtual camera and you can render a photorealistic image of the scene from any viewpoint — including views never actually photographed. Introduced by Mildenhall and colleagues in 2020, NeRF set off an explosion in neural 3D representations.

Concretely, the network is a multilayer perceptron (a plain fully-connected MLP). Its input is five numbers: the 3D position (three coordinates) and the viewing direction (two angles), hence '5D radiance field.' Its output is a color (red, green, blue) and a single density value — how much light that point blocks or emits. Density depends only on position (it is a property of the geometry), while color is allowed to depend on viewing direction too, which lets NeRF reproduce view-dependent effects like specular highlights and glossy reflections that change as you move around.

NeRF is trained per scene, with no 3D supervision at all — only a set of ordinary photos with known camera poses (typically obtained from structure-from-motion). For each training pixel, the system shoots the corresponding ray, samples points along it, asks the MLP for their colors and densities, and composites them with volume rendering into a single predicted pixel color. The only loss is the squared difference between this rendered color and the real pixel. Because the entire rendering process is differentiable, gradient descent pushes the network until its internal field reproduces all the training views — and, remarkably, interpolates correctly to new ones. A key trick, positional encoding, maps the raw coordinates through high-frequency sinusoids so the MLP can represent sharp detail rather than only smooth blur.

Vanilla NeRF is slow to train (hours) and slow to render, and follow-up work attacked both. Instant-NGP replaces much of the MLP with a learned multi-resolution hash grid, cutting training to seconds or minutes. Mip-NeRF handles aliasing by reasoning about cones rather than infinitely thin rays. Other variants add unbounded scenes (mip-NeRF 360), deformation for dynamic scenes, and few-image or generalizable training. NeRF's conceptual legacy is huge, and its main present-day rival for speed and editability is 3D Gaussian splatting.