Volume Rendering, NeRF, and 3D Gaussian Splatting

A. Neural Volume Rendering (60 points)

1. Differentiable Volume Rendering

1.3. Ray sampling (5 points)

Visualization

Grid
Grid
Rays
Rays

1.4. Point sampling (5 points)

Visualization

Sample points
Sample points

1.5. Volume rendering (20 points)

Visualization

Volume rendering
Volume rendering
Depth
Depth

2. Optimizing a basic implicit volume

2.2. Loss and training (5 points)

After training, the box center is (0.25, 0.25, 0.00) and the side lengths are (2.01, 1.50, 1.50).

2.3. Visualization

Optimized box
Optimized box
After training, view 0
After training, view 0
After training, view 1
After training, view 1
After training, view 2
After training, view 2
After training, view 3
After training, view 3

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

Visualization

NeRF, lego
NeRF, lego

Trained on the lego scene for 250 epochs at 128x128. The final training MSE was 0.0032.

4. NeRF Extras (CHOOSE ONE! More than one is extra credit)

4.1 View Dependence (10 bonus points)

Materials, view dependent
Materials, view dependent

The viewing direction is encoded with a low frequency embedding and concatenated to the 128 dim feature vector at the input of the color head.

4.3 VolSDF (10 bonus points)

Geometry
Geometry
Colour
Colour

Alpha sets the density inside the object, and beta sets how quickly that density falls to zero across the surface.

  1. A high beta spreads density over a thick band around the surface, which biases the SDF toward smooth shapes that lose thin parts. A low beta makes density close to a step function, which pushes the zero level set to sit exactly on the observed surface.
  2. A high beta is easier to train because points far from the current surface still get useful gradients. With a low beta, gradients only exist in a thin shell that the ray samples can step over.
  3. A low beta is more likely to give an accurate surface because the rendering weights concentrate at the zero level set. With a high beta, many different SDFs render to similar images, so the surface comes out blurrier.

Hyperparameters: I kept alpha = 10 and beta = 0.05, and trained for 250 epochs with a learning rate of 5e-4. Beta = 0.05 is still wider than the sample spacing of about 0.03, so the surface band gets sampled.

I compared three runs.

The default init starts the SDF near zero everywhere, which leaves surface fragments in regions no camera sees. Starting from a sphere gives a valid SDF that training only has to carve down, and the stronger eikonal term keeps empty space clean.

Command: implicit_function.geometric_init=True training.eikonal_weight=0.1

B. 3D Gaussian Splatting (60 points)

5. 3D Gaussian Rasterization (35 points)

5.5 Perform Splatting

Pre-trained chair
Pre-trained chair

6 Training 3D Gaussian Representations (15 points)

6.2 Perform Forward Pass and Compute Loss

Training progress (top: render, bottom: ground truth)
Training progress (top: render, bottom: ground truth)
Final renders
Final renders
Parameter Learning rate
opacities (pre-activation) 0.05
scales (log) 0.005
colours 0.02
means 0.0005

7 Extensions (Choose at least one! More than one is extra credit)

7.1 Rendering Using Spherical Harmonics (10 points)

5.5: DC colour only
5.5: DC colour only
7.1: full spherical harmonics
7.1: full spherical harmonics
View 3: without SH (left), with SH (right)
View 3: without SH (left), with SH (right)
View 12: without SH (left), with SH (right)
View 12: without SH (left), with SH (right)
View 20: without SH (left), with SH (right)
View 20: without SH (left), with SH (right)

Left: DC term only (Q5.5). Right: full spherical harmonics (Q7.1).