Neural 3D: Learned Depth, NeRF & Gaussian Splatting

Depth from a Single Image — Learned

Back in the very first guide of this track we hit a hard wall: a single photograph cannot tell you metric depth. The pinhole camera collapses a whole 3D ray onto one pixel, so a toy car held close and a real car far away can paint exactly the same pixels. Geometry alone is stuck — and that is precisely why we needed two views (stereo) and many views (SfM). Yet you, looking at one ordinary snapshot, instantly sense what is near and what is far. How?

You are not measuring — you are remembering. Your brain leans on learned cues: lines that converge toward a vanishing point (perspective), texture that grows finer in the distance (texture gradient), an object whose familiar real size betrays its distance, and the fact that nearer things occlude farther ones. None of these come from triangulation; they come from a lifetime of seeing the world. Deep networks do exactly the same trick, and we call it single-image depth prediction.

Concretely, we collect an enormous dataset of images each paired with a known depth map (from a depth sensor, a stereo rig, or a LiDAR scan). A convolutional or transformer network is trained so that, given the RGB image alone at test time, it outputs a per-pixel depth. The network is not solving geometry equation by equation; it has absorbed a statistical prior — "images that look like this usually have depth like that." This whole family of tasks is depth estimation, and the single-image, learned variant is its most magical-seeming member.

# Monocular depth at inference: one RGB image in, one depth map out.
# No second camera, no triangulation — just a learned prior.
img = load_rgb("holiday_photo.jpg")        # H x W x 3
depth = depth_net(img)                      # H x W, predicted (often RELATIVE) depth

# Scale ambiguity: depth is correct up to an unknown factor s.
# If we know ONE true distance (e.g. a measured object), we can anchor it:
s = true_distance_to_marker / depth[marker_y, marker_x]
metric_depth = s * depth                    # now in metres (approximately)

The strengths are obvious and delightful: it works on a single casual phone photo, no rig, no second viewpoint, no camera motion. The failure modes are equally instructive. A learned prior is only as good as what it has seen — show it a scene unlike anything in training (an aerial shot, a microscope image, a mirror, an Escher staircase) and it can confidently hallucinate wrong depth. Geometry never lies but often cannot answer; learned depth always answers but can be confidently wrong. Keep both truths in mind as we move from predicting depth to rendering whole worlds.

Representing a Scene as a Function: NeRF

Until now every 3D representation we met was explicit: a point cloud (a bag of dots), or a mesh (dots stitched into triangles) from Guide 4. You could point at a vertex and say "that piece of the world lives here." Neural Radiance Fields throw that intuition out and replace it with something radical: store the scene not as stuff, but as a function. A small neural network is the scene.

Picture a magic, glowing fog filling the room. You can walk up to any point in empty-or-occupied space, look in any direction, and ask two questions: "What colour am I seeing right here?" and "How solid — how opaque — is this spot?" A neural radiance field is a function that answers exactly those two questions for every point and every viewing direction. Train it well and it has memorised an entire 3D scene inside a few megabytes of network weights — a strikingly compact alternative to a dense mesh for 3D reconstruction.

The whole scene, compressed into one little function.

Let us read this symbol by symbol. The input on the left is a 5D query: \mathbf{x}=(X,Y,Z) is a 3D point in space (where you are standing in the fog), and \mathbf{d} is a unit vector giving the viewing direction (which way you are looking). The function F_{\Theta} is a small multilayer perceptron whose behaviour is fixed by its learned weights \Theta — those weights are the memorised scene. The output is \mathbf{c}=(r,g,b), the colour emitted toward you at that point, and \sigma, the volume density — loosely, how much opaque matter is packed at \mathbf{x}. A concrete read: query \mathbf{x}= the tip of a vase, \mathbf{d}= looking from the front; out comes \mathbf{c}= a glazed cream-white and a large \sigma (solid). Query a point in mid-air and \sigma\approx 0 (empty, light passes straight through).

Notice a deliberate asymmetry baked into the design: colour \mathbf{c} depends on both position and direction, while density \sigma depends on position only. That is physics, not a quirk. Whether matter exists at a point cannot change with where you look — a wall is solid no matter your angle. But the colour you see can change with angle: a glossy teapot throws a bright specular highlight that slides across its surface as you move. Letting \mathbf{c} be direction-dependent is exactly what lets NeRF capture shiny, view-dependent appearance that a plain coloured mesh cannot. We now have a function that describes the fog; the next section shows how to turn it into an actual picture by marching rays through it.

Volume Rendering: Turning the Field into an Image

We have a function that knows colour and density everywhere, but a function is not a photo. To get a single pixel we borrow the camera model from Guide 1 and reverse it: instead of asking where a world point lands on the sensor, we shoot a ray out from the camera centre through that pixel into the scene. Then we sample many points along the ray, query the NeRF at each for its colour and density, and composite them into one final colour. This recipe is called volume rendering, and it is the mathematical heart of NeRF — so we will take it slowly.

The key new idea is transmittance: how much light survives the journey from a point back to the camera. Imagine staring through layers of fog. Light emitted by something deep inside has to pass through everything in front of it, and each dense patch it crosses absorbs a little more. So a point that is dense (it glows strongly) but sits behind other dense stuff contributes almost nothing — it is occluded. A point that is dense and has clear air in front of it contributes a lot. Transmittance is the bookkeeping that makes nearer surfaces correctly hide farther ones, automatically.

The volume-rendering integral: every point's colour, weighted by how solid it is and how much light reaches it.

Read it left to right. The ray is \mathbf{r}(t)=\mathbf{o}+t\,\mathbf{d}: start at the camera origin \mathbf{o} and walk a distance t in viewing direction \mathbf{d}; the integral runs from a near bound t_n to a far bound t_f. At each depth t the NeRF gives density \sigma(\mathbf{r}(t)) and colour \mathbf{c}(\mathbf{r}(t),\mathbf{d}). The factor T(t) is the transmittance — the fraction of light that survives from the camera out to distance t. It is e^{-(\text{total density passed through so far})}: if you have already crossed a lot of \sigma, the exponent is very negative and T is near 0, so anything beyond is dimmed to nothing. Multiply these three — T(t) (does light get here?), \sigma (is there matter here?), \mathbf{c} (what colour is it?) — integrate along the ray, and you get C(\mathbf{r}), the final colour of that one pixel.

The same idea as a computable sum over N sample points along the ray.

A computer cannot do a true integral, so we slice the ray into N samples and turn it into a weighted sum. For sample i, \delta_i is the gap to the next sample, \sigma_i its density, and \mathbf{c}_i its colour. The term \alpha_i = 1-e^{-\sigma_i\delta_i} is that sample's opacity — read it as "how solid is this little slab?" If \sigma_i\delta_i is large, \alpha_i\to 1 (fully opaque, like a wall); if it is near 0, \alpha_i\to 0 (transparent air). The term T_i=\prod_{j<i}(1-\alpha_j) is the discrete transmittance: multiply the see-through-ness (1-\alpha_j) of every slab in front. So each sample's weight T_i\,\alpha_i reads as exactly "how visible \times how solid." Tiny worked example: three samples with \alpha=(0.1,\,0.9,\,0.8). The first contributes weight 1\times0.1=0.1; the second (1-0.1)\times0.9=0.81; the third (1-0.1)(1-0.9)\times0.8=0.072. The middle surface dominates, and the third — hiding behind the opaque second — barely registers. That is occlusion, computed for free.

Training a NeRF

Where does the scene come from? You hand the system a set of ordinary photos of an object or room, taken from many angles — and, crucially, the camera pose of each one (its position and orientation in space). A lovely tie-back: those poses come from exactly the pipeline of Guides 1 and 4 — camera calibration to get the intrinsics, and Structure-from-Motion to recover where each shot was taken. Classical geometry does the surveying; the neural field does the rendering. They are partners, not rivals.

1. Pick a batch of pixels from the training photos. For each, build its camera ray using the known pose and intrinsics (Guide 1).
2. Sample points along each ray and query the NeRF for colour and density at every point.
3. Composite with the differentiable volume renderer to get a predicted pixel colour \hat{C}(\mathbf{r}).
4. Compare \hat{C} to the real pixel colour with a photometric loss, then backpropagate to nudge the weights \Theta.
5. Repeat for hundreds of thousands of iterations until the renders match the photos from every view.

Photometric loss: make the rendered pixels match the real ones.

This loss is almost embarrassingly simple, which is part of its beauty. The sum runs over a batch \mathcal{R} of rays — one ray per training pixel. For each ray, \hat{C}(\mathbf{r}) is the colour our volume renderer produced and C_{\text{gt}}(\mathbf{r}) is the ground-truth colour of that pixel in the actual photo. The double bars \lVert\cdot\rVert_2^2 mean "squared distance between the two RGB colours" — e.g. predicted (0.8,0.2,0.2) versus true (0.9,0.1,0.2) gives (0.1)^2+(0.1)^2+0^2=0.02. Add that up over all sampled pixels and minimise it. Because every step from \Theta to \hat{C} is differentiable, gradient descent can flow the error all the way back into the weights. The astonishing part: there is no 3D supervision at all — no one ever tells the network the true geometry. It infers the entire 3D structure purely from the demand "render to look like these 2D photos from these viewpoints."

One ingredient quietly decides whether the result is crisp or a muddy blur: positional encoding. If you feed the raw coordinates (X,Y,Z) straight into the MLP, it produces oversmooth, washed-out scenes — networks have a built-in bias toward low-frequency, slowly-varying functions, so they cannot represent a sharp edge or fine texture. The fix is to lift each coordinate into a bank of high-frequency sine and cosine features before the network sees it, giving it the raw materials to express rapid change.

3D Gaussian Splatting: Fast & Explicit

Now meet the current state of the art, which flips NeRF's central choice on its head. NeRF is implicit and slow: the scene lives invisibly inside network weights, and you pay a heavy per-ray network query for every pixel. 3D Gaussian splatting goes back to an explicit representation — but a clever one. The scene is a cloud of millions of tiny 3D blobs, each a soft fuzzy ellipsoid carrying its own position, shape, colour, and opacity. Nothing implicit, no MLP to evaluate at render time.

One 3D Gaussian: a soft ellipsoidal blob centred at μ, shaped by its covariance Σ.

This is the same bell-shaped Gaussian you have met before, now in 3D. Here \mathbf{x} is any query point, \boldsymbol{\mu} is the blob's centre — its 3D position in the scene — and \Sigma is a 3\times3 covariance matrix that sets the blob's size, shape, and orientation (think of it as a stretched, tilted ellipsoid rather than a perfect ball). The expression (\mathbf{x}-\boldsymbol{\mu})^{\top}\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}) is a squared distance from the centre, but warped by \Sigma^{-1} so it counts distance differently along the ellipsoid's long and short axes. The \exp(-\tfrac12 \cdot) then makes the blob's influence G(\mathbf{x}) equal to 1 exactly at the centre and fade smoothly to 0 as you move away — densest in the middle, soft at the edges, no hard boundary. Beyond \boldsymbol{\mu} and \Sigma, each Gaussian also stores a colour (often view-dependent, so it too can do specular highlights) and an opacity. Millions of these together are the scene.

Why is it so fast? Rendering is rasterization, not ray marching. Recall from Guide 1 that a camera projects a 3D point to a 2D pixel; here we project each whole Gaussian — \Sigma included — down to a 2D ellipse on the image. That projected, screen-space ellipse is the "splat." GPUs are built to draw millions of such 2D primitives blazingly fast. We sort the splats by depth and blend them front-to-back with the very same alpha-compositing rule from volume rendering: \hat{C}=\sum_i T_i\,\alpha_i\,\mathbf{c}_i, where each \alpha_i now comes from the Gaussian's own opacity times its falloff at that pixel. No per-pixel network call anywhere — which is exactly why it renders in real time, hundreds of frames per second, where NeRF crawls.

How are the Gaussians found? By the very same idea that trained NeRF: differentiable rendering plus a photometric loss. You start from a sparse point cloud (helpfully, the SfM points from Guide 4), place a Gaussian at each, render, compare to the photos, and gradient-descend on every Gaussian's position, covariance, colour, and opacity. The twist is adaptive density control: during training the optimizer clones and splits Gaussians in under-reconstructed regions that need more detail, and prunes ones that have gone nearly transparent or useless. The set grows and shrinks itself to fit the scene. The whole thing is best pictured as painting a scene with countless soft, coloured, semi-transparent fuzzy dots — lay down enough of them and the picture sharpens into photorealism.

# Sketch of a Gaussian-splatting training step (one batch of pixels).
for step in range(num_iters):
    cam = sample_training_camera()           # known pose from SfM (Guide 4)
    splats_2d = project_gaussians(gaussians, cam)   # 3D ellipsoid -> 2D screen ellipse
    splats_2d.sort(key=depth)                # front-to-back
    pred = alpha_blend(splats_2d)            # same compositing as volume rendering
    loss = photometric(pred, cam.photo)      # ||pred - ground_truth||^2 (+ SSIM term)
    loss.backward()                          # differentiable splatting
    optimizer.step()                         # update mu, Sigma, colour, opacity
    gaussians = densify_and_prune(gaussians) # clone/split where blurry, drop the useless

The Future of 3D Vision

Stand back and see the whole landscape we have crossed. Classical geometry (Guides 1–4) is interpretable, metric, and frugal: from a handful of images it recovers real distances in real units, and you can audit every step — a vanishing point, an epipolar line, a triangulated point. Its weaknesses are appearance and density: it struggles with shiny view-dependent surfaces and does not, by itself, give you a photoreal renderable world. Neural methods (this guide) are the mirror image: NeRF and Gaussian splatting deliver jaw-dropping photorealism and smooth novel views, but they are hungry for many images, heavy on compute, and harder to interpret or measure.

The headline is that they are complementary, not competing. The two halves of this guide literally hand off to each other: modern photoreal systems begin with classical Structure-from-Motion to estimate camera poses and a starter point cloud, then feed those into a neural radiance field or Gaussian-splat model for rendering. Geometry surveys the world; neural fields paint it. The best pipelines today refuse to choose between them.

Where is the field heading? Several fronts are moving fast at once. Real-time and on-device 3D: Gaussian splatting already runs in browsers and on phones, putting captured spaces in your pocket. Generative 3D: models that turn a line of text or a single image into a full 3D asset, marrying the diffusion ideas from the 2D-generation track with the renderers you just learned. Dynamic and editable scenes: extending these representations across time for moving people and objects, and making them as easy to edit as a 3D model. And the application pull is enormous — robots that need a metric, navigable map; AR/VR that demands photoreal worlds at 90 frames per second; autonomous vehicles fusing geometry and learning to understand a street in three dimensions.