16-825 Learning for 3D Vision

Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Andrew ID: djonnala

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

xy_gridrays
Pixel grid Rays

1.4 Point Sampling

Sample points along rays

1.5 Volume Rendering

ColorDepth
Box color render Box depth render

The image appears to be split horizontally, with a small phase difference between the two halves. This comes from the chunk size in the config. Setting the chunk size to W*H matched the TA output.

2. Optimizing a basic implicit volume

2.2 Loss and Training

xyz
Box Center0.250.250.00
Box Side Length2.011.511.52

2.3 Visualization

The visual matches the reference, except for the horizontal split and phase difference from chunking noted in 1.5.

Optimized box

3. Optimizing a Neural Radiance Field

I trained for about 50 epochs and got a result that matches the reference image. I used the network from the NeRF paper as a reference and implemented it from scratch. I first forgot the sigmoid on color, so the render stayed black. After adding it, the network trained well.

Low-resolution NeRF

4.1 View Dependence

As in the original NeRF paper, the model takes the direction vector d and concatenates it with the output of the first 8 linear layers to predict color.

Initialization mattered here. An early high-resolution run produced a very soft bulldozer after 10 epochs, but it did not checkpoint. When training started again, the model produced a black image the whole time because the density stayed at 0. Kaiming initialization, plus a bias of 0.1 on the density head only, made training much more stable.

High-resolution view-dependent NeRF
NeRF on high-resolution images with view dependence, 450 epochs.
Without View DependenceWith View Dependence
Without view dependence With view dependence

The view-dependent render reacts to the viewing direction and reflects light toward the camera. The model without view dependence does not change color with orientation.

B. 3D Gaussian Splatting

5.5 Perform Splatting

Gaussian splatting render
Some high-frequency noise appears only at certain angles.

6.2 Forward Pass and Loss

Number of iterations: 1000. Mean PSNR: 27.157. Mean SSIM: 0.910.

HyperparametersValue
iterations1000
means lr0.0001
opacity lr0.005
scales lr0.005
colors lr0.001
quatsnot trained (isotropic)
Training progress
Training progress
Final truck renders
Final renders

7.2 Training on a Harder Scene

For the best result, I used:

IterationsPSNRSSIMFinal RendersTraining Progress
10000 23.577 0.865 10k final renders 10k training progress
HyperparametersValue
iteration10000
means lr0.005 with decay scheduler
opacity lr0.005 with warmup scheduler
scales lr0.005 with warmup scheduler
colors lr0.001 with warmup scheduler
quats lr0.0005 with warmup scheduler
python train_harder_scene.py --gaussians_per_splat 3000 --num_itrs 10000

Extra ablations for 7.2

Training for more iterations

IterationsPSNRSSIMFinal Renders
200017.6010.632 2k final renders
400018.2260.661 4k final renders

Learning rate scheduling

python train_harder_scene.py --gaussians_per_splat 3000 --num_itrs 4000 --use_isotropic
IterationsPSNRSSIMFinal Renders
100017.3880.600 Isotropic 1k final renders
400017.9890.664 Isotropic 4k final renders
IterationsPSNRSSIMFinal Renders
100017.3880.600 High mean-lr final renders
IterationsPSNRSSIMFinal Renders
200021.0880.712 High mean-lr 2k final renders