Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Author: Po-Ting Ko

Andrew ID: potingk

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

Pixels are mapped to NDC [−1, 1] and unprojected with camera.unproject_points(..., from_ndc=True); rays start at the camera center.

xy grid Rays
Grid Rays

1.4 Point sampling

Uniform depths in [near, far], jittered within each bin during training. Samples of the first camera (every 16th ray):

Sample points

1.5 Volume rendering

αi = 1 − exp(−σi Δti),    Ti = ∏j<i (1 − αj),    C = Σi Ti αi ci,    D = Σi Ti αi ti
Color Depth (normalized by its maximum)
Part 1 Depth

2. Optimizing a basic implicit volume

Random rays: a random permutation of all pixels, keeping the first n_pixels. Loss: MSE on colors.

Center Side lengths
Box (0.25, 0.25, 0.00) (2.01, 1.50, 1.51)

Part 2

The result matches the TA reference.

3. Optimizing a Neural Radiance Field (NeRF)

Part 3

4. NeRF Extras

All Part 4 runs use TF32. Test PSNR is the mean over the 200 test views.

4.1 View Dependence

Materials, view dependent

Ground truth (left) and prediction (right) on test views:

Materials test views

Trade-off between view dependence and generalization (128 × 128)

Model Train PSNR Test PSNR Train − test
A: no view dependence 23.46 23.09 0.37
B: direction, 2 frequencies 24.29 23.66 0.63
C: direction, 8 frequencies 24.39 23.71 0.68

View dependence comparison

4.2 Coarse/Fine Sampling

On one ray, all 128 fine samples land within [2.98, 3.15] around the surface:

Coarse weights and fine samples along a ray

Model Queries per ray Test PSNR Train time Render time per image
Single network, 128 samples (Part 3) 128 24.21 11.7 min 145 ms
Single network, 192 samples 192 24.51 15.3 min 212 ms
Coarse (64) + fine (64 + 128) 256 24.43 19.6 min 286 ms

Coarse/fine comparison

Single, 128 Single, 192 Coarse/fine
Single 128 Single 192 Coarse/fine

4.3 VolSDF

σ(x) = α Ψβ(−d(x)),    Ψβ = CDF of a Laplace(0, β) distribution

Results

Lego, 250 epochs, α = 10, β = 0.05 (best); test PSNR 22.29:

Geometry Color
VolSDF geometry VolSDF color

Effect of β (rows: β = 0.01, 0.05, 0.2):

Beta comparison

β Test PSNR Zero level set − rendered depth Weight outside the surface
0.01 22.24 −0.098 7%
0.05 22.29 −0.063 25%
0.2 21.04 +0.070 49%

1. How does high β bias the learned SDF? What about low β? With a high β, half of the color comes from points outside the object. The network compensates by moving the zero level set inwards (+0.07 behind the rendered depth), giving a shrunken, smoothed surface where parts merge. With a low β, the density is concentrated at d = 0, so the SDF is nearly unbiased but sharp. The remaining offset comes from α, not β.

2. Would an SDF be easier to train with volume rendering and low β or high β? Why? High β. The gradient only flows through samples within a few β of the surface, so with a high β every ray gets a signal even when the surface is far from the object. Without geometric initialization, β = 0.2 still trains (21.39), but β = 0.01 fails (13.18):

No geometric initialization

3. Would you be more likely to learn an accurate surface with high β or low β? Why? Low β. The rendered color then comes from d = 0 itself, so the photometric loss constrains the surface directly, as the 7% weight outside the surface and the sharper mesh show. This works once training succeeds, for example with geometric initialization.

Settings. - β = 0.05 and α = 10 give the best PSNR with a clean surface. - Geometric initialization is what makes β ≤ 0.05 trainable. - Softplus keeps the SDF gradient smooth for the eikonal loss.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

All 4 unit tests pass.

Part 5 render

Note on W. The code uses W = R, as the unit tests and the course staff recommend. However, PyTorch3D uses row vectors (Xview = X R + T), so the true world-to-view rotation is Rᵀ. I checked this with Monte Carlo sampling: the covariance of projected samples matches W = Rᵀ, not W = R. The two agree for frontal views, but from oblique views W = R smears flat Gaussians. This does not affect isotropic Gaussians (Part 6).

W = R vs W = R^T

6. Training 3D Gaussian Representations

Parameter Learning rate
opacities (pre-sigmoid) 0.05
colours 0.02
scales (log) 0.005
means 0.0005
Iterations Mean PSNR Mean SSIM
1000 29.70 0.943

Part 6 training progress

Part 6 final renders

7. Extensions

7.1 Rendering Using Spherical Harmonics

Q5.5: DC only Q7.1: full SH
DC only With SH

DC vs SH comparison

7.2 Training on a Harder Scene

Materials scene, 128 × 128, random initialization. The baseline is the exact Q6 setup (isotropic Gaussians, Q6 learning rates, L1 loss, 1000 iterations).

Method Iterations #Gaussians Mean PSNR Mean SSIM
Baseline (Q6 setup) 1000 10,000 18.40 0.676
Improved 3000 8,216 25.62 0.914

Last frame of each progress GIF (baseline, improved, ground truth):

Baseline vs improved

Baseline training progress Improved training progress
Baseline progress Improved progress
Baseline final renders Improved final renders
Baseline final renders Improved final renders

Modifications (compared to Q6):

Comparison. - Baseline. The Gaussians stay near the origin: at lr 0.0005 they can move only about 0.5. As a result, the outer spheres become large, blurry blobs. - Improved. The Gaussians find every sphere within a few hundred iterations. Pruning first removes Gaussians in empty space (10,000 → 5,426), then densification adds them back where detail is needed. - Caveat. There is no separate ablation per change.