Assignment 3

16-825 Learning for 3D Vision

Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Andrew ID: pburujwa

Name: Parth Burujwale


Implementation Details

Training and rendering were performed on an NVIDIA H100 GPU through PSC Bridges-2.


A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel Grid

Pixel Grid

Ray Directions

Ray Directions

1.4 Point Sampling

Sampled Points Along Rays

1.5 Volume Rendering

Spiral Rendering

Volume Rendering of a Box

Depth Visualization

Normalized Depth Map

2. Optimizing a Basic Implicit Volume

2.2 Loss and Training

Optimized Parameters

Parameter Value
Box Center (0.25, 0.25, 0.00)
Side Lengths (2.01, 1.50, 1.50)

2.3 Visualization

Optimized Box

3. Optimizing a Neural Radiance Field

Experimental Settings

Setting Value
Dataset Lego
Image Resolution 128 × 128
Training Epochs 250
Rays per Batch 1,024
Samples per Ray 128
Position Harmonic Frequencies 6
MLP Layer Setting 6
Hidden Width 128
Input Skip Setting Layer Index 3
Initial Learning Rate 0.0005
Loss Mean Squared Error

Result

Lego Neural Radiance Field

4. NeRF Extensions

4.1 View Dependence

Experimental Settings

Setting Value
Dataset Materials
Image Resolution 128 × 128
Training Epochs 250
Rays per Batch 1,024
Samples per Ray 128
Position Harmonic Frequencies 6
Direction Harmonic Frequencies 2
Spatial Hidden Width 128
Direction Branch Hidden Width 64
Initial Learning Rate 0.0001
Loss Mean Squared Error

Result

View Dependent Materials NeRF

View Dependence and Generalization

View dependence allows the network to represent reflections and other appearance changes across viewing directions. However, a highly expressive color branch can also memorize training views instead of learning an appearance model that generalizes.

I kept density independent of viewing direction and used fewer harmonic frequencies for direction than position. The direction branch also has a smaller hidden width. These choices limit its capacity, but a controlled evaluation on held out views would be needed to measure their effect on generalization.


4.2 Coarse and Fine Sampling

The coarse network evaluates uniformly sampled points along each ray. Its rendering weights define a probability distribution for sampling additional points. These weights are detached before inverse CDF sampling.

The additional points are combined with the original samples and sorted by depth. The fine network evaluates this combined set. The training loss is the sum of the coarse and fine RGB reconstruction losses.

Experimental Settings

Setting Value
Dataset Lego
Image Resolution 128 × 128
Training Epochs 250
Coarse Samples per Ray 64
Additional Fine Samples per Ray 64
Total Samples Evaluated by Fine Network 128
Rendering Chunk Size 1,024

Result

Coarse and Fine NeRF Rendering

Speed and Quality Comparison

Hierarchical sampling directs additional samples toward regions that contribute strongly to the image. This can improve the sampling of surfaces without increasing the uniform sampling density everywhere.

However, it requires a second network and an additional rendering pass. The observed training rate was approximately 96 iterations per second, compared with approximately 175 iterations per second for the earlier Part 3 run. The architectures also differed, so this comparison does not isolate the cost of hierarchical sampling alone.

The resulting reconstruction remained consistent across views, but the visual improvement over the earlier NeRF was modest. The combined coarse and fine loss is not directly comparable to the single network RGB loss.


4.3 VolSDF

Alpha and Beta

Alpha controls the density scale. Increasing alpha makes the interior more opaque over a given distance.

Beta controls the width of the density transition around the surface. A larger beta produces a broader transition, while a smaller beta produces a sharper transition near the zero level set.

How does high beta bias the learned SDF? What about low beta?

High beta spreads density across a wider region around the surface. This can allow the model to explain images using a soft transition rather than a sharply localized boundary.

Low beta concentrates density changes near the surface. This supports sharper boundaries, but makes optimization more sensitive to the surface position and the sampled points.

Would an SDF be easier to train with low beta or high beta? Why?

A relatively high beta is generally easier during early training because a wider region provides useful rendering gradients.

With low beta, the transition becomes narrow and samples may miss it. This can make it harder to update an incorrectly positioned surface.

Would an accurate surface be more likely with high beta or low beta? Why?

A suitably low beta can produce a more sharply localized surface once the geometry is reasonably initialized.

However, sufficient sampling and successful optimization are still necessary. A very low beta can hurt training if the transition is poorly sampled.

Hyperparameter Comparison

Setting Baseline Selected Configuration
Alpha 10.0 10.0
Beta 0.05 0.025
Samples per Ray 128 256
Eikonal Weight 0.02 0.02
Training Epochs 250 250
Initial Learning Rate 0.0005 0.0005
Image Resolution 128 × 128 128 × 128

Baseline Color Rendering

Baseline VolSDF Color

Baseline Geometry

Baseline VolSDF Geometry

Selected Color Rendering

Selected VolSDF Color

Selected Geometry

Selected VolSDF Geometry

Discussion

The selected configuration produced cleaner color outlines and fewer floating streaks. Its geometry still contains rough thin structures and some merged components.

Reducing beta sharpens the density transition, while increasing the number of samples helps resolve that narrower transition. Since both settings changed together, the improvement should be attributed to the combined configuration.


B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

5.2 Evaluating 2D Gaussians

I evaluated the Gaussian exponent using the displacement between each pixel and Gaussian center, together with the inverse covariance matrix.

The supplied unit tests reported:

4/4 Tests Passed


5.5 Splatting

Pretrained Chair Rendering

Pretrained Chair Gaussian Rendering

6. Training 3D Gaussian Representations

6.1 Parameters and Optimizer

Parameter Learning Rate
Means 0.001
Scales 0.005
Colors 0.01
Opacities 0.05

The smaller means learning rate limits large changes to the point cloud initialization. Colors and opacities can adapt more quickly to image supervision.


6.2 Training and Evaluation

Setting Value
Optimizer Adam
Training Iterations 1,000
Loss L1
Held Out Test Views 5

Quantitative Results

Mean PSNR = 29.813 dB

Mean SSIM = 0.942

Training Progress

The top row shows predictions during training. The bottom row shows ground truth images.

Truck Training Progress

Final Renderings

Truck Final Renderings

7. Gaussian Splatting Extensions

7.1 Rendering Using Spherical Harmonics

Without Directional Spherical Harmonics

Chair Without Directional SH

With Spherical Harmonics

Chair With Spherical Harmonics

Comparison 1: Frame 000

Spherical Harmonics Comparison Frame 000

With SH, the green backrest is darker and the gold floral pattern has stronger contrast. Without directional SH, the upholstery appears brighter and flatter.

Comparison 2: Frame 008

Spherical Harmonics Comparison Frame 008

With SH, the darker green regions contrast more strongly with the gold pattern. Streaks around the chair’s edges remain visible in both versions.

Comparison 3: Frame 016

Spherical Harmonics Comparison Frame 016

With SH, the central backrest becomes darker while the upper region remains brighter. This creates more variation in shading, while the chair’s outline remains unchanged.


7.2 Training on a Harder Scene

Experimental Settings

Setting Baseline Improved
Gaussian Count 10,000 10,000
Gaussian Type Isotropic Isotropic
Initialization Random Random
Initial Means Standard Deviation 0.2 0.8
Initial Scales Starter random scales, approximately 0 to 0.01 0.03
Initial Opacity Approximately 0.9997 0.5
Means Learning Rate 0.0001 0.001
Scales Learning Rate 0.005 0.005
Colors Learning Rate 0.01 0.01
Opacities Learning Rate 0.05 0.05
Training Iterations 1,000 5,000
Image Resolution 128 × 128 128 × 128
Loss L1 L1
Optimizer Adam Adam

Quantitative Results

The provided script evaluates these models on the held out validation split.

Configuration Mean PSNR ↑ Mean SSIM ↑
Baseline 17.671 dB 0.654
Improved 22.628 dB 0.823
Improvement 4.957 dB 0.169

Baseline Training Progress

Materials Baseline Training Progress

Improved Training Progress

Materials Improved Training Progress

Baseline Final Renderings

Materials Baseline Final Renderings

Improved Final Renderings

Materials Improved Final Renderings

Modifications and Discussion

I made the following changes compared with the baseline:

The broader initialization distributes Gaussians across a larger region. Larger initial scales increase their projected coverage, while lower initial opacity reduces the dominance of individual Gaussians during early training.

The higher means learning rate allows centers to move farther from their initial positions. Longer training provides additional opportunities to fit the scene.

The baseline contains blurred objects, incomplete coverage, and sparse detail. The improved configuration separates objects more clearly and reconstructs their colors and arrangement more successfully. Fine reflective details remain rough.

The improved configuration increased both PSNR and SSIM. However, several settings and the training duration changed together, and the runs did not use matched random initialization. Therefore, this comparison measures the combined configuration rather than the individual effect of each modification.

Both runs retained isotropic Gaussians and L1 loss. I did not use adaptive densification, anisotropic covariance optimization, or spherical harmonic coefficient training for this experiment.


References