16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3. Ray sampling

Both outputs match the TA output:

xy_grid rays
xy_grid Ray directions

1.4. Point sampling

Samples from the first camera are visualized here below.

point_samples

1.5. Volume rendering

Depth are normalized by its maximum value.

Rendering Depth
rendered_box normalized_depth

2. Optimizing a basic implicit volume

The box centre and side lengths after optimization:

Centre: (0.25, 0.25, 0.00), Side lengths: (2.01, 1.50, 1.50)

The resulting rectangular box rendering matches the TA reference.

optimized_box

3. Optimizing a Neural Radiance Field

Results on the test set:

PSNR: 24.1958 dB; SSIM: 0.8836.

I implemented a six-layer MLP with 128 hidden units, ReLU density output, and Sigmoid RGB output. The visualization is as follows:

Lego NeRF

4. NeRF Extras

4.1 View Dependence

Modification on top of Part 3: I added a color network that takes the viewing direction as an additional input. The position network (adapted from Part 3) predicts density and features. The color network combines these features with the encoded viewing direction to predict color.

I compared this view-dependent approach with the original view-independent NeRF from Part 3 on the Materials scene. The results are as follows:

NeRF model Test PSNR (dB) Test SSIM gif rendering
Position only 23.0487 0.8470 no_view_dependence
Position + direction 23.6961 0.8613 view_dependence

Adding view dependence increases PSNR by 0.6474 dB and SSIM by 0.0143, showing some improvement. In the renderings, we can also see brighter regions on some materials.

The trade-offs between increased view dependence and generalization quality are as follows: Increased view dependence can better represent areas with reflections and specular highlights (non-diffuse effects). However, excessive view dependence could cause the model to memorize and overfit to training views, thereby degrading generalization to unseen viewpoints. The current results do not show degraded novel-view generalization performance.

4.2 Coarse/Fine Sampling

My coarse/fine sampling strategy is as follows: The coarse network evaluates 128 uniform samples and produces a sampling distribution, and 128 additional depths are sampled accordingly and merged with the coarse samples. A separate fine network evaluates all 256 points. Both networks are supervised with MSE loss on rendered images vs. ground-truth images.

NeRF model Test PSNR (dB) Test SSIM Training iters/s
w/o Coarse/fine sampling (Part 3) 24.1958 0.8836 139.32
w/ Coarse/fine sampling (Part 4.2) 24.5522 0.8884 53.46

Visualization of the results from Coarse/fine sampling: Coarse/fine Lego NeRF

With coarse/fine sampling, PSNR improves by 0.3564 dB and SSIM by 0.0048, and training iterations take approximately 2.6× longer. Coarse/fine sampling uses 384 network queries per ray, compared with 128 queries for the baseline method.

4.3 VolSDF

Training configuration: alpha=10, beta=0.05, Eikonal weight 0.02, learning rate 5e-4, and 250 epochs on Lego images.

Results on the test set:

PSNR: 21.0314 dB; SSIM: 0.7552.

Color Extracted geometry
color geometry

The main vehicle is recognizable. However, the extracted surface contains floating artifacts and "overlapped" layers.

Results of changing beta:

BetaMean PSNR (dB)Mean SSIM
0.0220.90640.7946
0.0521.16540.7603
0.1017.50590.5596

This comparison is done by rendering the previously trained epoch-200 checkpoint with different beta values, and the results are averaged over the first five validation views. beta=0.05 has the highest PSNR and the relative high SSIM in this comparison, which supports my chosen setting.

Explanation of alpha and beta: alpha defines the interior density and affects the opacity scale. beta controls the width of the transition from exterior to interior density.

  1. High beta vs. low beta: High beta widens the density transition region around the surface, which is more suitable for and biased towards thicker and blurrier boundaries. Low beta narrows the density transition around the surface, encouraging sharper surface boundaries.
  2. Training difficulty: High beta is easier to train with volume rendering as the rendering gradients are spread over a wider region. Low beta could be harder to optimize.
  3. Surface accuracy: With a low beta and appropriate sampling, the model may localize a surface more accurately, since the density transition region is narrower.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

A GIF rendering visualization is as follows:

rasterization

6 Training 3D Gaussian Representations

Final selected learning rate:

Parameter Selected learning rate
Position 1e-4
Scale 5e-3
Opacity 5e-3
Color 5e-3

A smaller learning rate for position can prevent the drifting of the gaussians.

Comparison on the learning rate of color:

Color learning rateTest PSNR (dB)Test SSIM
1e-326.8900.898
5e-3 28.9060.927

Both experiments were trained for 1,000 iterations.

The final run yields clearer windows and wheels in the visualization. training_progress

truck_renderings

7 Extensions

7.1 Rendering Using Spherical Harmonics

Only DC spherical harmonic components (Part 5.5) All spherical harmonics components (Part 7.1)
only_dc with_SH
Image Only DC Full SH
0 Frame 0 without SH Frame 0 with SH The backrest appears to be darker and clearer with full SH, with the gold patterns more distinct to notice. The gold patterns could be more specular and shiny, so it may be better captured with view-dependent full SH.
9 Frame 9 without SH Frame 9 with SH The backrest and cushion are darker and more clearly defined with full SH.
12 Frame 12 without SH Frame 12 with SH The cushion looks darker and its gold pattern has stronger contrast with full SH. The backrest also shows clearer shading variations.