CMU 16-825 Learning for 3D Vision

Assignment 3:
Volume Rendering, NeRF, and 3D Gaussian Splatting

Sichen Guo

Task 1 /

Differentiable Volume Rendering

1.3   Ray Sampling

Pixel grid visualization
Pixel grid visualization
World-space ray visualization
World-space ray visualization

1.4   Point Sampling

Sampled points from the first camera
Sampled points from the first camera

1.5   Volume Rendering

Spiral rendering of the box
360-degree rendering of the box
Depth map normalized by its maximum value
Depth map normalized by its maximum value
Task 2 /

Optimizing a Basic Implicit Volume

2.3   Visualization

Spiral rendering of the optimized box ta
360-degree rendering of the optimized box from ta
Spiral rendering of the optimized box
360-degree rendering of the optimized box

Box center: (0.25, 0.25, 0.00)
Box side lengths: (2.00, 1.50, 1.50)

Task 3 /

Optimizing a Neural Radiance Field

Visual Results

360-degree rendering of the Lego NeRF
360-degree rendering of the Lego NeRF without view dependence
Task 4 /

NeRF Extras

4.1   View Dependence

360-degree rendering of the Lego NeRF without view dependence
360-degree rendering of the Lego NeRF without view dependence
360-degree rendering of the Lego NeRF with view dependence
360-degree rendering of the Lego NeRF with view dependence

Materials scene with view dependence
Materials scene with view dependence

Adding view dependence produces subtle appearance changes in the Lego scene while preserving similar overall geometry. View dependence is particularly relevant to the materials scene, where reflections and highlights vary with viewing direction. However, greater directional flexibility can overfit the training views and reduce consistency at unseen angles. Limiting this flexibility helps balance realistic appearance with generalization to novel views.

Task 5 /

3D Gaussian Rasterization

5.5   Perform Splatting

Pre-trained Gaussians rendered using view-independent color
Pre-trained Gaussians rendered using view-independent color
Task 6 /

Training 3D Gaussian Representations

6.2   Forward Pass, Loss, and Training Results

I used Adam with learning rates of 0.0002 for means (positions), 0.005 for scales, 0.05 for opacities, and 0.0025 for colors. The model was configured to train for 1,000 iterations.

PSNR (dB): 29.493
SSIM: 0.940

Toy truck: final renderings
Toy truck: final renderings
Toy truck: training progress and ground truth
Toy truck: training progress and ground truth
Task 7 /

Gaussian Splatting Extensions

7.1   Rendering Using Spherical Harmonics

GIF Comparison:

Without SH: DC component only (same result as 5.5)
Without SH: DC component only (same result as 5.5)
With SH: view-dependent rendering
With SH: view-dependent rendering

Matched View 1:

View 1: DC only
View 1: DC only
View 1: with SH
View 1: with SH

In the DC-only rendering, the upper half of the chair cushion is noticeably bright. With SH, the entire cushion appears darker and more shaded, including the upper-right corner.

Matched View 2:

View 2: DC only
View 2: DC only
View 2: with SH
View 2: with SH

In the DC-only rendering, the cushion remains noticeably bright. From this viewpoint, the cushion in the SH rendering becomes brighter overall, including the upper-right corner.

Matched View 3:

View 3: DC only
View 3: DC only
View 3: with SH
View 3: with SH

In the DC-only rendering, the upper half of the cushion remains noticeably bright. From this viewpoint, the SH rendering shows uneven brightness across the cushion: the upper-right corner is darker, while the other areas are relatively bright.

Across all three comparisons, the chair’s texture also appears clearer in the SH renderings.

7.2   Training on a Harder Scene

Baseline Results:

Baseline final rendering results
Baseline training progress

Improved Results:

Baseline training progress
Improved training progress

Since both the baseline and improved models were trained for 30,000 iterations, their training progress GIFs are large, so we uniformly sample 100 frames from each.

Improved final renderings along a 360-degree orbit

Baseline:
PSNR: 17.737 dB
SSIM: 0.674

Improved:
PSNR: 24.933 dB
SSIM: 0.903

Improved Settings:

Compared with the baseline, I made the following modifications:

Training duration and learning-rate scheduling: Both the baseline and improved models were trained for 30,000 iterations, compared with 1,000 iterations for the toy-truck experiment. For the improved model, the position learning rate decays exponentially from 0.0005 to 0.000005 over 30,000 steps, allowing larger updates early and finer adjustments later. The learning rates for opacities and scales remain 0.05 and 0.005.

Anisotropic Gaussians: I replaced isotropic Gaussians with anisotropic Gaussians, optimizing three scales and a rotation for each Gaussian to better fit scene geometry. The rotation learning rate is 0.001.

View-dependent appearance: I added trainable spherical harmonics to model appearance changes across viewing directions. The DC coefficients use a learning rate of 0.0025, while higher-order coefficients use 0.000125.

Combined reconstruction loss: I replaced the L1-only loss with 0.8 × L1 + 0.2 × (1 − SSIM) to encourage both pixel accuracy and structural similarity.

Improved initialization: Relative to the original initialization, I increased the standard deviation of random positions from 0.2 to 0.5, initialized opacity to 0.1 instead of nearly 1, and initialized scales using the root mean squared distance to each point's three nearest neighbors. These changes provide broader initial coverage, reduce excessive occlusion, and adapt Gaussian sizes to local point spacing.

Comparison With NeRF:

NeRF reconstruction results
Improved Gaussian Splatting reconstruction results

The improved Gaussian representation captures surface cracks and holes, with more pronounced gloss and view-dependent color changes. Visually, it reconstructs these details more clearly than my NeRF result.