Assignment 3: Volume Rendering, NeRF & 3D Gaussian Splatting

Learning for 3D Vision

Name: Arnav Samal

Andrew ID: asamal

Late days used: 2

1. Differentiable Volume Rendering

This part implements camera ray generation, stratified sampling along rays, and emission-absorption volume rendering.

Ray and sample visualization

Rendered color and depth

2. Optimizing a Basic Implicit Volume

Random pixel sampling and mean squared error were used to optimize the position and side lengths of an implicit box from posed training images.

Optimized box rendering
Optimized box rendering Spiral visualization after training.

Learned box parameters

Box center (x, y, z) (0.25, 0.25, 0.00)
Side lengths (x, y, z) (2.01, 1.50, 1.50)

3. Optimizing a Neural Radiance Field (NeRF)

A positional-encoding MLP predicts density and RGB color at sampled 3D positions. The network is trained with image supervision using mean squared error.

NeRF rendering of the Lego scene
Lego NeRF Spiral rendering of the trained Lego scene.

4. NeRF Extension - View Dependence

The view-dependent color branch uses viewing direction as an input to color prediction. This allows the model to represent directional appearance effects such as highlights and reflections.

View-dependent NeRF rendering of the Materials scene
Materials scene · View-dependent NeRF Spiral rendering of the Materials scene.

View dependence makes the model more expressive because it can represent appearance changes caused by viewing direction, such as highlights and reflections. However, this added flexibility can also increase the risk of overfitting to the training views, potentially reducing generalization to unseen viewpoints. A view-independent model is simpler, while a view-dependent model can represent more complex appearance effects but may require sufficient training views to generalize well.

5. 3D Gaussian Rasterization

The rasterizer projects 3D Gaussian means and covariances into screen space, evaluates per-pixel alpha, computes front-to-back transmittance, and blends color, depth, and silhouette contributions.

Pretrained 3D Gaussian rasterization animation
Pretrained Gaussian scene View sequence rendered using the Gaussian rasterizer.

6. Training 3D Gaussian Representations

Gaussians were optimized against posed training images using Adam and an L1 image loss.

Training iterations 1,000
Mean PSNR 28.431 dB
Mean SSIM 0.929

Optimizer settings

Parameter group Learning rate
Gaussian means 0.0001
Gaussian scales 0.005
Gaussian colors 0.01
Gaussian opacities 0.01

7.1. Spherical Harmonics

This section compares 3D Gaussian Splatting renderings with and without spherical harmonics. The rendering from Question 5.5 uses the original view-independent color representation, while Question 7.1 includes spherical harmonics for view-dependent color.

Rendering Comparison

RGB Image Comparisons

Each pair compares the same camera view rendered without and with spherical harmonics.

Comparison 1 - View 1

In this front view, the rendering with spherical harmonics has richer, darker green tones and more noticeable brown fabric details. Without spherical harmonics, the chair appears lighter and more uniform. The overall shape and structure remain nearly identical, and the depth and silhouette maps show no significant visible differences.

Comparison 2 - View 2

From this angled view, the rendering with spherical harmonics shows slightly richer colors and more contrast in the green fabric and brown decorative details. The rendering without spherical harmonics appears somewhat lighter. Both versions preserve the same chair geometry and viewpoint, with very similar depth and silhouette maps.

Comparison 3 - View 3

In this view, the two renderings are visually very similar. The chair retains the same overall structure and surface appearance, with only subtle differences in color and texture between the two versions. The depth and silhouette maps are also nearly identical, suggesting that the main visible differences come from color representation rather than changes in geometry.