Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting
Joshua Tsai | andrew id: joshuat3
We randomly sample pixels from training images and generate the corresponding camera rays.
We optimized the box parameters using an RGB reconstruction loss (MSE) for 1000 epochs.
Optimized Box Center:
(0.25, 0.25, 0.00)
Optimized Side Lengths:
(2.02, 1.52, 1.51)
We implemented a position-based NeRF MLP with harmonic positional encoding to predict volume density and RGB color. The model was trained on the LEGO Bulldozer dataset for 250 epochs.
We extended the NeRF model to predict color using both 3D position and viewing direction, while keeping density dependent only on position. We compared view-independent and view-dependent models on the Materials dataset.
View dependence allows NeRF to model direction-dependent appearance, including specular highlights and reflections. However, increasing view dependence also increases model flexibility and the risk of overfitting to training viewpoints, potentially reducing generalization to unseen views.
The view-independent baseline is simpler and can reconstruct the overall geometry and colors, but it cannot explicitly represent view-dependent appearance.
We implemented a simplified 3D Gaussian rasterizer with 3D-to-2D projection, depth sorting, and front-to-back alpha compositing.
We optimized isotropic 3D Gaussians to reconstruct a toy truck using multi-view RGB images. The Gaussian means were initialized from the provided point cloud, and the parameters were optimized using Adam with an L1 reconstruction loss.
| Parameter | Learning Rate |
|---|---|
| Means | 0.0001 |
| Scales | 0.001 |
| Colours | 0.005 |
| Opacities | 0.005 |
Training Iterations: 1000
Mean PSNR: 28.110 dB
Mean SSIM: 0.929
We extended our 3D Gaussian rasterizer to support degree-3 spherical harmonics (SH) for view-dependent color rendering. Unlike the DC-only representation used in Section 5.5, which assigns a fixed color to each Gaussian, the full SH representation computes color based on the viewing direction.
The following GIFs compare the DC-only and full SH renderings of the same pre-trained Gaussian scene. Both methods use identical Gaussian geometry, scales, and opacities. The only difference is how Gaussian colors are computed.
We selected three camera viewpoints to compare the RGB renderings side by side. In each comparison, the left image uses DC-only colors, while the right image uses full spherical harmonics.
Full SH produces noticeable differences in the shading and color contrast of the chair's green backrest and seat. The geometric structure remains unchanged, while the higher-order SH coefficients introduce viewing-direction-dependent appearance.
From this elevated viewpoint, the full SH rendering shows different green tones and brightness patterns on the chair's fabric. In particular, the backrest and seat exhibit changes in local shading compared with the DC-only rendering.
Both renderings preserve the chair's overall geometry and detailed structure. However, full SH produces differences in the color and brightness around the backrest and armrests, illustrating the additional view-dependent appearance information captured by SH.
Overall, spherical harmonics allow each Gaussian to represent color variations across viewing directions, which cannot be modeled using only the DC component. The differences are especially visible in the chair's fabric shading and local brightness patterns.
However, the differences are relatively subtle in some views. Since both methods use the same pre-trained Gaussian geometry, the primary changes are in appearance rather than shape. These visual comparisons demonstrate the effect of view-dependent color modeling, but do not establish a quantitative improvement in reconstruction accuracy.