16-825 Learning for 3D Vision

Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting
Joshua Tsai | andrew id: joshuat3

Part 1: Volume Rendering

1.3 Ray Sampling

1.4 Point Sampling

1.5 Volume Rendering

Part 2: Optimizing a Box

2.1 Random Ray Sampling

We randomly sample pixels from training images and generate the corresponding camera rays.

2.2 Loss and Training

We optimized the box parameters using an RGB reconstruction loss (MSE) for 1000 epochs.

Optimized Box Center: (0.25, 0.25, 0.00)
Optimized Side Lengths: (2.02, 1.52, 1.51)

2.3 Visualization

Part 3: Optimizing a NeRF

We implemented a position-based NeRF MLP with harmonic positional encoding to predict volume density and RGB color. The model was trained on the LEGO Bulldozer dataset for 250 epochs.

Part 4: NeRF Extras

4.1 View Dependence

We extended the NeRF model to predict color using both 3D position and viewing direction, while keeping density dependent only on position. We compared view-independent and view-dependent models on the Materials dataset.

Discussion

View dependence allows NeRF to model direction-dependent appearance, including specular highlights and reflections. However, increasing view dependence also increases model flexibility and the risk of overfitting to training viewpoints, potentially reducing generalization to unseen views.

The view-independent baseline is simpler and can reconstruct the overall geometry and colors, but it cannot explicitly represent view-dependent appearance.

Part B: 3D Gaussian Splatting

5. 3D Gaussian Rasterization

We implemented a simplified 3D Gaussian rasterizer with 3D-to-2D projection, depth sorting, and front-to-back alpha compositing.

Gaussian Splatting RGB, Depth and Silhouette
Gaussian Splatting RGB, Depth and Silhouette

6. Training 3D Gaussian Representations

We optimized isotropic 3D Gaussians to reconstruct a toy truck using multi-view RGB images. The Gaussian means were initialized from the provided point cloud, and the parameters were optimized using Adam with an L1 reconstruction loss.

6.1 Training Configuration

Parameter Learning Rate
Means 0.0001
Scales 0.001
Colours 0.005
Opacities 0.005

Training Iterations: 1000

6.2 Evaluation Results

Mean PSNR: 28.110 dB
Mean SSIM: 0.929

Training Progress

3D Gaussian Training Progress
Training progress: rendered predictions (top) and ground truth (bottom).

Final Rendering

3D Gaussian Final Rendering
Final toy truck reconstruction after 1000 iterations.

7.1 Rendering Using Spherical Harmonics

We extended our 3D Gaussian rasterizer to support degree-3 spherical harmonics (SH) for view-dependent color rendering. Unlike the DC-only representation used in Section 5.5, which assigns a fixed color to each Gaussian, the full SH representation computes color based on the viewing direction.

Rendering Comparison

The following GIFs compare the DC-only and full SH renderings of the same pre-trained Gaussian scene. Both methods use identical Gaussian geometry, scales, and opacities. The only difference is how Gaussian colors are computed.

DC-only Gaussian Splatting Rendering
DC-only rendering (view-independent colors).
Full Spherical Harmonics Gaussian Splatting Rendering
Full SH rendering (degree-3, view-dependent colors).

RGB Comparisons

We selected three camera viewpoints to compare the RGB renderings side by side. In each comparison, the left image uses DC-only colors, while the right image uses full spherical harmonics.

Frame 005 — Front View

Frame 005: DC-only vs Full SH
Frame 005: DC-only (left) and Full SH (right).

Full SH produces noticeable differences in the shading and color contrast of the chair's green backrest and seat. The geometric structure remains unchanged, while the higher-order SH coefficients introduce viewing-direction-dependent appearance.

Frame 012 — High-Angle View

Frame 012: DC-only vs Full SH
Frame 012: DC-only (left) and Full SH (right).

From this elevated viewpoint, the full SH rendering shows different green tones and brightness patterns on the chair's fabric. In particular, the backrest and seat exhibit changes in local shading compared with the DC-only rendering.

Frame 018 — Oblique Front View

Frame 018: DC-only vs Full SH
Frame 018: DC-only (left) and Full SH (right).

Both renderings preserve the chair's overall geometry and detailed structure. However, full SH produces differences in the color and brightness around the backrest and armrests, illustrating the additional view-dependent appearance information captured by SH.

Discussion

Overall, spherical harmonics allow each Gaussian to represent color variations across viewing directions, which cannot be modeled using only the DC component. The differences are especially visible in the chair's fabric shading and local brightness patterns.

However, the differences are relatively subtle in some views. Since both methods use the same pre-trained Gaussian geometry, the primary changes are in appearance rather than shape. These visual comparisons demonstrate the effect of view-dependent color modeling, but do not establish a quantitative improvement in reconstruction accuracy.