Assignment 3 — Neural Volume Rendering and Gaussian Splatting

16-825: Learning for 3D Vision · Haotian (hzhan2)

1. Differentiable Volume Rendering

1.3 Ray sampling

I put the pixel grid at depth 1 in NDC, unproject it to world space, and subtract the camera center to get normalized ray directions.

Pixel grid
Pixel grid (xy_grid)
Ray directions
Ray directions
Q1.3 comparison with reference
Comparison with the reference images

1.4 Point sampling

64 evenly spaced depths between 0 and 5; each point is origin + depth × direction. Below are the samples from the first camera.

Sample points
Sample points along rays
Q1.4 comparison with reference
Comparison with the reference image

1.5 Volume rendering

Each sample’s weight is its opacity times the transmittance of the samples in front of it (T = 1 for the first one). The depth map is normalized by its max.

alpha_i  = 1 - exp(-sigma_i * delta_i)
T_i      = prod_{j<i} (1 - alpha_j)
color    = sum_i T_i * alpha_i * c_i
depth    = sum_i T_i * alpha_i * z_i
Part 1 spiral render
Spiral rendering
Depth map
Depth map

2. Optimizing a Basic Implicit Volume

Each step I train on 4,096 random pixels (shuffle the grid, keep the first n_pixels) with MSE loss, Adam at lr 0.0005 for 1,000 epochs. Final box, rounded to 0.01:

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Before training
Before training (view 0)
After training
After training (view 0)
Optimized box spiral
Optimized box, spiral rendering

Other training views

Before training view 1
Before training (view 1)
After training view 1
After training (view 1)
Before training view 2
Before training (view 2)
After training view 2
After training (view 2)
Before training view 3
Before training (view 3)
After training view 3
After training (view 3)

3. Optimizing a Neural Radiance Field (NeRF)

6-frequency harmonic embedding, 6-layer 128-wide MLP with a skip at layer 3, ReLU for density and Sigmoid for color. Trained on lego at 128×128 for 250 epochs, 1,024 rays and 128 samples per ray. I ran it twice and both look about the same.

NeRF lego
NeRF on lego, 128×128
NeRF lego second run
NeRF on lego, 128×128 (second run, default output name)

4.1 View Dependence

The color head also gets the ray direction (2-frequency embedding). Density only depends on position, so geometry stays consistent across views.

Trade-off: more view dependence can capture the specular highlights on the spheres, but the network can also overfit by explaining each training image with view-specific color, which gives floaters and flicker at new views. Less view dependence generalizes better but makes shiny things look matte.

Materials 128
Materials, 128×128 (final loss ~0.0044)
Materials 400
Materials, 400×400 (final loss ~0.0014)

4.2 Coarse/Fine Sampling

A coarse NeRF uses 64 stratified samples. I use its weights as a PDF to draw 128 more samples, and a fine NeRF renders all 192. Loss = MSE(coarse) + MSE(fine).

Lego, 128×128, 100 held-out views, epoch 200:

ModelTrain speedRender time / imageVal PSNR
Single network, 128 samples41 it/s146 ms24.55 dB
Single network, 192 samples26 it/s223 ms24.76 dB
Coarse/fine, 64 + 128 samples22 it/s300 ms24.80 dB

Speed vs. quality: coarse/fine is the slowest (two networks per ray) but gets the best PSNR, since more samples land near the surface. At this resolution the gain is small. (The 192-sample run used seed 2; seed 1 collapsed to black.)

Coarse fine
Coarse/fine, 64 + 128 samples
Single 192
Single network, 192 samples

4.3 VolSDF

6-layer 128-wide MLP for distance (no output activation) plus a small color head. Eikonal loss weight 0.02. Density = α · Ψβ(−d), the Laplace CDF from the VolSDF paper.

Intuition for alpha and beta

α is the density inside the object. β is how wide the transition from empty to solid is around the surface; smaller β means a sharper surface.

  1. How does beta bias the SDF? High β gives a blurry, inflated surface and loses thin parts. Low β forces the zero level set to sit right on the surface.
  2. Easier to train? High β, because points near the surface still get gradients. With low β gradients mostly vanish, which is why my β = 0.01 run failed.
  3. More accurate surface? Low β (if it trains), since the color has to come from points right on the surface.

Hyper-parameter experiments (lego, α = 10, 250 epochs)

βFinal lossResult
0.20.011Trains, but color is blurrier and the mesh is blobby, with a thick slab of floor geometry.
0.050.008Best. Sharp color and a recognizable bulldozer mesh, with a few floaters under the base.
0.010.039Failed. Density never formed, the render is mostly black, and marching cubes found an empty mesh.

Best: α = 10, β = 0.05. Sharp enough for a clean surface, but still wide enough to train from scratch.

Best result (β = 0.05)

Geometry beta 0.05
Geometry
Color beta 0.05
Color

Other beta values

Geometry beta 0.2
β = 0.2, geometry
Color beta 0.2
β = 0.2, color
Geometry beta 0.01
β = 0.01, geometry (failed)
Color beta 0.01
β = 0.01, color (failed)

5. 3D Gaussian Rasterization

I project each 3D Gaussian to 2D, sort by depth, and alpha-blend front to back. Color, depth and mask all use the same weights. The 4 unit tests pass. Below is the pre-trained chair with DC color only (RGB | depth | mask), 2,048 Gaussians per splat.

Q5 render
Output GIF of render.py
All 32 frames of the Q5 render (RGB | depth | mask)
the Q5 render frame 0
Frame 0
the Q5 render frame 1
Frame 1
the Q5 render frame 2
Frame 2
the Q5 render frame 3
Frame 3
the Q5 render frame 4
Frame 4
the Q5 render frame 5
Frame 5
the Q5 render frame 6
Frame 6
the Q5 render frame 7
Frame 7
the Q5 render frame 8
Frame 8
the Q5 render frame 9
Frame 9
the Q5 render frame 10
Frame 10
the Q5 render frame 11
Frame 11
the Q5 render frame 12
Frame 12
the Q5 render frame 13
Frame 13
the Q5 render frame 14
Frame 14
the Q5 render frame 15
Frame 15
the Q5 render frame 16
Frame 16
the Q5 render frame 17
Frame 17
the Q5 render frame 18
Frame 18
the Q5 render frame 19
Frame 19
the Q5 render frame 20
Frame 20
the Q5 render frame 21
Frame 21
the Q5 render frame 22
Frame 22
the Q5 render frame 23
Frame 23
the Q5 render frame 24
Frame 24
the Q5 render frame 25
Frame 25
the Q5 render frame 26
Frame 26
the Q5 render frame 27
Frame 27
the Q5 render frame 28
Frame 28
the Q5 render frame 29
Frame 29
the Q5 render frame 30
Frame 30
the Q5 render frame 31
Frame 31

6. Training 3D Gaussian Representations (truck)

Isotropic Gaussians are initialized from the truck point cloud, and I train their means, scales, opacities and colors with Adam and an L1 loss.

SettingValue
GaussiansIsotropic, initialized from the provided point cloud
Learning rates (Adam)means 0.0002, scales 0.001, opacities 0.01, colours 0.01
LossL1 between rendered and ground-truth image
Iterations1000
Mean PSNR28.378
Mean SSIM0.930
Q6 training progress
Training progress (top: prediction, bottom: ground truth)
Q6 final renders
Final renders of the training views

7.1 Rendering Using Spherical Harmonics

Color now uses all SH coefficients (degree 3), evaluated along the camera-to-Gaussian direction.

Q5 DC only
Q5.5: DC component only
Q7.1 with SH
Q7.1: full spherical harmonics

Side-by-side comparisons (left: DC only, right: full SH, same frame)

Comparison frame 0
Frame 0 (front view)

SH gives much more contrast on the fabric: darker folds and a clearer gold pattern. DC-only looks flat and washed out.

Comparison frame 8
Frame 8 (top view)

From the top, the cushions are darker and the shading on the backrest shows its curve. The trim looks about the same.

Comparison frame 16
Frame 16

Same as frame 0: shadows on the fabric are darker and the embroidery brighter. DC is just an average color, so it can’t show this.

Comparison frame 24
Frame 24 (underside view)

From below they look almost the same. Depth and mask never change, since SH only affects color.

All 32 frames of the Q7.1 render (RGB | depth | mask)
the Q7.1 render frame 0
Frame 0
the Q7.1 render frame 1
Frame 1
the Q7.1 render frame 2
Frame 2
the Q7.1 render frame 3
Frame 3
the Q7.1 render frame 4
Frame 4
the Q7.1 render frame 5
Frame 5
the Q7.1 render frame 6
Frame 6
the Q7.1 render frame 7
Frame 7
the Q7.1 render frame 8
Frame 8
the Q7.1 render frame 9
Frame 9
the Q7.1 render frame 10
Frame 10
the Q7.1 render frame 11
Frame 11
the Q7.1 render frame 12
Frame 12
the Q7.1 render frame 13
Frame 13
the Q7.1 render frame 14
Frame 14
the Q7.1 render frame 15
Frame 15
the Q7.1 render frame 16
Frame 16
the Q7.1 render frame 17
Frame 17
the Q7.1 render frame 18
Frame 18
the Q7.1 render frame 19
Frame 19
the Q7.1 render frame 20
Frame 20
the Q7.1 render frame 21
Frame 21
the Q7.1 render frame 22
Frame 22
the Q7.1 render frame 23
Frame 23
the Q7.1 render frame 24
Frame 24
the Q7.1 render frame 25
Frame 25
the Q7.1 render frame 26
Frame 26
the Q7.1 render frame 27
Frame 27
the Q7.1 render frame 28
Frame 28
the Q7.1 render frame 29
Frame 29
the Q7.1 render frame 30
Frame 30
the Q7.1 render frame 31
Frame 31

7.2 Training on a Harder Scene (materials)

Both runs use init_type="random" with 10,000 Gaussians, evaluated on 100 views at 128×128. The baseline is my unchanged Q6 setup.

Baseline (Q6 setup)Improved
Iterations10002500
Learning ratesmeans 0.0002, scales 0.001, opacities 0.01, colours 0.01means 0.002 → 0.00002 (exponential decay), scales 0.005, opacities 0.05, colours 0.01, quaternions 0.001
LossL10.8 · L1 + 0.2 · (1 − SSIM)
Gaussians10,000 isotropic10,000 → 25,499 anisotropic (densified)
Mean PSNR16.29324.372
Mean SSIM0.6160.891

Baseline

Q7.2 baseline training progress
Training progress (top: prediction, bottom: ground truth)
Q7.2 baseline final renders
Final renders

The baseline only learns a blob in the middle: the random points start near the origin and the means learning rate is too small to reach the objects.

Improved

Q7.2 improved training progress
Training progress (top: prediction, bottom: ground truth)
Q7.2 improved final renders
Final renders

Modifications in the best method (compared to Q6)

  1. Better init: points spread uniformly in a radius-1.2 ball, starting opacity 0.1, scales from nearest-neighbour distance.
  2. Anisotropic Gaussians with learned rotations.
  3. Higher learning rates: means 0.002 decaying to 0.00002, opacities 0.05, scales 0.005, quaternions 0.001.
  4. Loss: 0.8 · L1 + 0.2 · (1 − SSIM), like the paper.
  5. Simple densification: from iteration 300 to 1800, clone or split Gaussians with large gradients, prune transparent or huge ones (max 30,000), and reset opacity at iteration 1000.
  6. 2500 iterations instead of 1000.

I also used gradient checkpointing in the renderer so training fits on my 8 GB GPU (doesn’t change results).