Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Name: Chenzhe Zhang

Course: 16-825 Learning for 3D Vision

1.3 Ray Sampling

Pixel Coordinates Ray Directions
Pixel Coordinates Ray Directions

1.4 Point Sampling

Sampled Points

1.5 Volume Rendering

Volume Rendering Normalized Depth
Volume Rendering Normalized Depth

2. Optimizing a Basic Implicit Volume

2.1 Random Ray Sampling

I randomly sample 4,096 pixels without replacement from each image and generate their corresponding rays for training to reduce GPU memory usage.

2.2 Loss and Training

Parameter x y z
Center 0.25 0.25 0.00
Side lengths 2.01 1.50 1.50

2.3 Visualization

Optimized Box

3. Neural Radiance Field

Lego NeRF

I trained a position-encoded MLP with six hidden layers of 128 units for 250 epochs at a resolution of 128 × 128.

4.1 View Dependence

View-dependent Materials NeRF

Density depends only on position, while RGB also uses the encoded viewing direction. This allows the model to represent view-dependent reflections, but excessive directional capacity can overfit the training views and reduce generalization to unseen views. I use a small color branch with 64 hidden units and train for 250 epochs at a resolution of 128 × 128.

5. 3D Gaussian Rasterization

5.1 Projecting 3D Gaussians

I compute the 3D covariance from each Gaussian's scale and rotation, then project it to 2D using the camera rotation and the projection Jacobian. Gaussian centers are projected to pixel coordinates.

5.2 Evaluating 2D Gaussians

I compute the Gaussian power at each pixel using its offset from the projected mean and the inverse 2D covariance.

All four provided unit tests passed: 4/4.

5.3 Filtering and Sorting Gaussians

I transform Gaussian centers into camera space, discard Gaussians with non-positive depth, and sort the remaining Gaussians from front to back.

5.4 Alphas and Transmittance

I compute per-pixel alpha by multiplying each Gaussian's opacity by its Gaussian response. Transmittance is the product of $(1-\alpha)$ over all preceding Gaussians and is carried across mini-batches.

5.5 Splatting

Gaussian Splatting Results

From left to right: RGB rendering, depth map, and silhouette mask.

6. Training 3D Gaussian Representations

6.1 Parameters and Optimizer

I optimize 10,000 isotropic Gaussians using Adam with the following learning rates.

Parameter Learning rate
Means 0.0001
Scales 0.005
Opacities 0.05
Colors 0.01

6.2 Training and Evaluation

I train for 1,000 iterations using an L1 image loss.

Metric Result
Mean PSNR 29.102 dB
Mean SSIM 0.937

Training Progress

Training Progress

Top row: rendered images. Bottom row: ground truth.

Final Renderings

Final Truck Renderings

Final renderings of the training views.

7. Gaussian Splatting Extension

7.1 Rendering Using Spherical Harmonics

I use spherical harmonics up to degree 3 to compute view-dependent RGB colors.

DC Only

DC Only

Full Spherical Harmonics

Full SH

Same-View RGB Comparisons

Left: DC only. Right: full spherical harmonics.

Frame 000

Frame 000 Comparison

Full SH produces darker green upholstery and stronger contrast in the gold pattern.

Frame 005

Frame 005 Comparison

Full SH changes the brightness across the seat and backrest, while the silhouette remains unchanged.

Frame 020

Frame 020 Comparison

The difference is smaller on the pale underside. Similar streaking artifacts remain around the legs in both versions.