16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Name: Wen Wu
Andrew ID: wengleiw

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel grid Rays
Pixel grid visualization Ray visualization

1.4 Point Sampling

Sample points from the first camera

1.5 Volume Rendering

Box spiral color rendering

Normalized depth visualization

2. Optimizing a Basic Implicit Volume

2.2 Loss and Training

Box center (x, y, z): (0.25, 0.25, 0.00)

Box side lengths (x, y, z): (2.01, 1.50, 1.50)

Before training After training
Box before training, camera 0 Box after training, camera 0

2.3 Visualization

Optimized box spiral rendering

3. Optimizing a Neural Radiance Field (NeRF)

Lego NeRF spiral rendering

4. NeRF Extensions (Choose One)

4.1 View Dependence

View-dependent NeRF results on the materials scene

View dependence versus generalization trade-offs: With view dependence, color can be predicted differently when viewpoint changes. This can lead to more realistic and diverse color rendering due to materials and changed reflection from different viewing angles. However, this can also lead to overfitting of view-specific color, i.e. less generalizable. If we ever test it using a new direction absent from training data, the color can deviate from what it should be.

4.2 Coarse/Fine Sampling

Coarse/fine sampling was added on top of the models from earlier sections; each run uses the same settings as its base model except for the renderer, which is switched from volume to hierarchical_volume:

Lego (Section 3 NeRF + coarse/fine) Materials (Section 4.1 view-dependent NeRF + coarse/fine)
Coarse/fine NeRF on the lego scene Coarse/fine view-dependent NeRF on the materials scene

Speed versus quality trade-offs: With Coarse/Fine Sampling, we are using one more MLP and more sample points. This means more GPU memory and compute usage during training and rendering. At the same time, since we give more and better point sampling at dense region, the rendering quality is improved, especially for locations with high frequency texture.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

5.5 Perform Splatting

Pre-trained Gaussian rendering

Gaussian RGB, depth, and mask frame

6. Training 3D Gaussian Representations

6.2 Training Results

Parameter Learning rate
Means 0.0005
Scales 0.005
Opacities 0.01
Colors 0.005

Training iterations: 1000

PSNR: 28.887

SSIM: 0.935

Truck training progress

Truck final renders

7. Gaussian Splatting Extensions (Choose at Least One)

7.1 Rendering Using Spherical Harmonics

DC component only (Section 5.5) All spherical harmonic components
DC-only rendering Spherical harmonic rendering

Comparison 1

DC component only All spherical harmonic components
Comparison 1, DC only Comparison 1, spherical harmonics

Observed differences: In comparison 1, the left image without SH shows a sudden change of color in the middle, while the right image with SH shows correct color based on the viewing direction. This means that without SH we can only learn an average fixed color, and with SH we can have different predicted color from different directions.

Comparison 2

DC component only All spherical harmonic components
Comparison 2, DC only Comparison 2, spherical harmonics

Observed differences: In comparison 2, we can clearly see the right image gives more details for the reflective texture compared to the left image. This means that with this viewing direction there should be reflection on those textures, while without SH we failed to predict that. In both cases, the difference is because SH takes into account the viewing direction that can lead to different color predictions. This gives us more realistic rendering since color can change if we move the camera, due to material and reflection.

7.2 Training on a Harder Scene

Parameter Baseline learning rate Improved learning rate
Means 0.0005 (same as Section 6.2) 0.0005
Scales 0.005 (same as Section 6.2) 0.005
Opacities 0.01 (same as Section 6.2) 0.01
Colors 0.005 (same as Section 6.2) 0.005
Rotations (quaternions) Not trained (isotropic) 0.001
Result Baseline Improved
Training iterations 1000 2000
PSNR 18.751 19.151
SSIM 0.662 0.693
Baseline training progress Improved training progress
Baseline materials training progress Improved materials training progress
Baseline final renders Improved final renders
Baseline materials final renders Improved materials final renders

Comparison of training progress, PSNR, and SSIM: PSNR increased by 0.4, and SSIM increased by 0.031. The quality difference is unnoticeable in the gif. The improved progress GIF only appears to converge faster because each frame covers 40 iterations instead of 20; per iteration, the two runs progress similarly.

All modifications to the Section 6.2 approach used by the best method: