16-825 · Learning for 3D Vision

Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Part A · Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel grid visualized as color: red increases to the left, green increases upward
xy grid
Absolute ray directions for the first camera visualized as RGB
Rays

1.4 Point Sampling

Sample points along the rays of the first camera, forming a wedge-shaped frustum
Sample points from the first camera

1.5 Volume Rendering

Rainbow-colored box rendered from 20 viewpoints around it
Color
Depth map of the box, normalized by its maximum
Depth, normalized by its maximum

2. Optimizing a Basic Implicit Volume

2.2 Loss and Training

Parameterxyz
Box center0.250.250.00
Side lengths2.011.501.50

Rounded to the nearest 0.01.

2.3 Visualization

Optimized rainbow-colored box rendered from 20 viewpoints around it
Optimized box

3. Optimizing a Neural Radiance Field (NeRF)

NeRF rendering of the lego bulldozer from 20 viewpoints around it
NeRF on the lego bulldozer

4. NeRF Extras

4.1 View Dependence

Materials scene rendered by the view-dependent NeRF from 20 viewpoints
With view dependence
Materials scene rendered by the NeRF without view dependence from 20 viewpoints
Without view dependence
Held-out test view of the material spheres: ground truth, without view dependence, with view dependence
Test view 107. Left to right: ground truth, without, with view dependence.
ModelTrain PSNRTest PSNRTrain − test
Without view dependence23.3823.100.28
With view dependence24.0723.530.54

nerf_materials_512.yaml, 512×512. PSNR (dB) over 40 training and 40 test views.

View dependence vs. generalization

The encoded ray direction is concatenated to the position feature before the color head; density depends on position only.

View dependence adds specular highlights on the glossy spheres and raises test PSNR by 0.43 dB, but training PSNR rises more (0.69 dB), so the train–test gap grows from 0.28 to 0.54 dB. The extra freedom lets the network explain inconsistencies between training views with view-dependent color instead of better geometry, which overfits the training cameras. A low-frequency direction encoding and a small color head limit this.

4.2 Coarse/Fine Sampling

Lego bulldozer at 512 by 512 rendered by a single NeRF from 20 viewpoints
Without coarse/fine
Lego bulldozer at 512 by 512 rendered with coarse/fine sampling from 20 viewpoints
With coarse/fine
Held-out lego test view: ground truth, without coarse/fine, with coarse/fine
Test view. Left to right: ground truth, without, with coarse/fine.
Without coarse/fineWith coarse/fine
Network evaluations per ray12864 coarse + 192 fine
Parameters107k215k
Test PSNR (dB)24.7025.29
Training step (ms)9.717.6
Training time (min)14.424.9
Render one 512×512 image (s)0.911.84

Lego, 512×512, 250 epochs, 10 position-encoding frequencies. PSNR over 20 test views; times on one RTX 4090.

Speed vs. quality

The coarse network renders 64 stratified samples; its weights form a pdf along the ray, from which 128 more depths are sampled, and the fine network renders all 192. The loss is coarse MSE + fine MSE.

Quality: test PSNR +0.59 dB and sharper edges, since the fine samples concentrate near surfaces. You can see finer lego details and sharper shadows in the "with coarse/fine" test image.

Speed: twice the network evaluations per ray and two networks, so rendering is 2.0× and training 1.7× slower.

4.3 VolSDF

Mesh extracted from the learned SDF of the lego bulldozer, from 20 viewpoints
Geometry
Lego bulldozer rendered by VolSDF from 20 viewpoints
Color

python -m volume_rendering_main --config-name=volsdf_surface renderer.alpha=20

What \(\alpha\) and \(\beta\) do

\(\sigma = \alpha\,\Psi_\beta(-d)\), where \(\Psi_\beta\) is the Laplace CDF. \(\alpha\) is the density inside the object (\(\sigma \to \alpha\) for \(d \ll 0\)). \(\beta\) controls how sharply the density rises across the surface: \(\sigma = \alpha/2\) at \(d = 0\) and ramps over about \(\pm 3\beta\). Small \(\beta\) approaches a hard surface; large \(\beta\) gives a fuzzy shell.

Questions

  1. How does high \(\beta\) bias your learned SDF? What about low \(\beta\)?

    High \(\beta\) spreads density over a wide band, so a smooth SDF whose zero level set is off the true surface can still match the images; thin parts vanish (\(\beta = 0.2\) loses the loader arms). Low \(\beta\) ties the zero level set to the visible surface, biasing the SDF toward sharp surfaces.

  2. Would an SDF be easier to train with volume rendering and low \(\beta\) or high \(\beta\)? Why?

    High \(\beta\): many samples fall in the wide band, so even a wrong SDF gets smooth, non-zero gradients. With low \(\beta\) only samples within a few \(\beta\) of the surface get gradients, and rays can step over it.

  3. Would you be more likely to learn an accurate surface with high \(\beta\) or low \(\beta\)? Why?

    Low \(\beta\), if the sampling can resolve it: the rendered surface coincides with the zero level set, so fitting the images places it accurately. High \(\beta\) allows a blurry or offset surface.

One frame from each sphere-initialized run: color on top, geometry below
Left to right: \((\alpha, \beta)\) = (10, 0.05), (10, 0.01), (10, 0.2), (20, 0.05). Top: color; bottom: geometry.
Run\(\alpha\)\(\beta\)Test PSNRMesh pieces (% of faces in largest piece)
Default, no sphere init100.0521.712,618 (81%)
Default100.0521.281 (100%)
Low \(\beta\)100.0121.543 (99.7%)
High \(\beta\)100.220.264 (90.5%)
Chosen200.0522.084 (99.7%)

Test PSNR (dB) over 20 views. Mesh pieces: connected components of the extracted mesh.

Chosen settings

Sphere initialization (\(d = \text{MLP}(\mathbf{x}) + \lVert\mathbf{x}\rVert - 1\)): without it all of space starts inside the object, so unseen regions become spurious surfaces (2,618 mesh pieces).

Low \(\beta\) (0.05) keeps the thin parts that high \(\beta\) (0.2) loses.

\(\alpha = 20\) (\(= 1/\beta\), as in VolSDF) makes rays opaque right at the surface and gives the best PSNR.

Part B · 3D Gaussian Splatting

5. 3D Gaussian Rasterization

Pre-trained chair Gaussians rendered from 32 viewpoints: color, depth and silhouette side by side
Left to right: color, depth, silhouette mask. Command: python render.py

6. Training 3D Gaussian Representations

ParameterLearning rate
Means0.001
Scales0.005
Colours0.0025
Opacities0.05

Iterations: 1,000 · PSNR: 30.15 · SSIM: 0.944

Training progress on four truck views: renders on top, ground truth below
Training progress, one frame every 20 iterations. Top: renders; bottom: ground truth.
Final renders of the trained truck Gaussians over all training views
Final renders of the training views. Command: python train.py

7. Extensions

7.1 Rendering Using Spherical Harmonics

Chair Gaussians rendered with spherical harmonics from 32 viewpoints: color, depth and silhouette
With spherical harmonics (7.1). Left to right: color, depth, silhouette mask.
Chair Gaussians rendered with only the DC color from 32 viewpoints: color, depth and silhouette
Without spherical harmonics, DC color only (5.5).

Comparison 1 · view 15

View 15 of the chair back: DC color on the left, spherical harmonics on the right
Left: without spherical harmonics. Right: with spherical harmonics.

Differences

With spherical harmonics the fabric is darker and more saturated, and the gold brocade in the center shows a bright metallic highlight. Without them the backrest is a flat, view-averaged green.

Comparison 2 · view 3

View 3 of the chair from above: DC color on the left, spherical harmonics on the right
Left: without spherical harmonics. Right: with spherical harmonics.

Differences

From above, the seat shows light and dark variation and the white frame gains shading; without spherical harmonics the chair looks evenly lit and flat. Depth and silhouette are unchanged.

7.2 Training On a Harder Scene

ParameterLearning rate
Means0.005, decayed exponentially to 0.00005
Scales0.005
Colours0.0025
Opacities0.05
Quaternions0.001

Iterations: 3,000 (baseline: 1,000)

MethodPSNRSSIM
Baseline (Q6 setup)18.610.692
Improved23.400.854
Baseline training progress on four materials views: renders on top, ground truth below
Baseline training progress. Top: renders; bottom: ground truth.
Improved training progress on four materials views: renders on top, ground truth below
Improved training progress. Top: renders; bottom: ground truth.
Final renders of the improved method over all materials training views
Improved final renders. Command: python train_harder_scene.py (baseline: --baseline)

Modifications

  • Initialization: the means are spread uniformly over the view (±1.3) instead of a ball of radius about 0.5 at the origin, so Gaussians reach the spheres at the edges. They start mostly transparent (opacity 0.1) and small (half the spacing to their neighbours); with near-opaque, larger Gaussians the spread initialization covered the image in fog and failed. This is the main gain.
  • Anisotropic Gaussians: scales per axis and trainable rotations.
  • Longer training with a means learning-rate schedule: 3,000 iterations, with the means lr decaying from 0.005 to 0.00005 so the Gaussians first move far, then settle.

The loss (L1) and the other learning rates are the same as Q6.