Assignment 3: Volume Rendering, Neural Radiance Field (NeRF), and Three-Dimensional (3D) Gaussian Splatting

Name: Yannick Daniel Gibson
Andrew identifier: ydg

A. Neural Volume Rendering (60 points)

1. Differentiable Volume Rendering

1.1. Familiarize yourself with the code structure

The pipeline connects camera rays, point sampling, an implicit scene, and volume rendering.

1.2. Outline of tasks

I implement ray sampling, point sampling, and compositing to render an implicit scene.

1.3. Ray sampling (5 points)

Camera rays map image pixels to viewing directions in world space.

1.4. Point sampling (5 points)

Uniform samples along each ray provide the points used for volume rendering.

1.5. Volume rendering (20 points)

Volume rendering combines sampled colors and densities into color and normalized depth images.

2. Optimizing a Basic Implicit Volume

2.1. Random ray sampling (5 points)

Random pixel samples provide batches of camera rays for training.

2.2. Loss and training (5 points)

Minimizing color mean squared error (MSE) recovers the box center (0.25, 0.25, 0.00) and side lengths (2.01, 1.50, 1.50).

2.3. Visualization

The optimized box reaches a training-view peak signal-to-noise ratio (PSNR) of 61.27 dB and structural similarity (SSIM) of 0.99998.

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

A 6-layer, 128-unit multilayer perceptron (MLP) with positional encoding learns the Lego scene for novel-view rendering.

Model PSNR (dB) SSIM
Lego NeRF 25.57 0.9000

4. NeRF Extras (choose one; more than one is extra credit)

4.1. View Dependence (10 bonus points)

View-dependent color changes with camera direction, allowing it to capture reflections that position-only color cannot, though this flexibility can also overfit; with matched training on 100 materials images at 400 x 400, view dependence changes held-out PSNR by +1.06 dB.

Model PSNR (dB) SSIM
Position only 24.27 0.8408
View dependent 25.33 0.8561
Position-only vs. view-dependent vs. ground truth
Position-only vs. view-dependent vs. ground truth

4.2. Coarse/Fine Sampling (10 bonus points)

Coarse/fine sampling combines 64 uniform and 128 importance samples per ray, compared with 192 uniform samples.

Model PSNR (dB) SSIM
Uniform 26.11 0.9179
Coarse and fine 26.11 0.9185

Coarse/fine sampling makes the red rear lights and roof-mounted warning beacon easier to distinguish, with a more vivid red color.

This small quality gain costs more computation: the coarse network evaluates 64 samples and the fine network evaluates all 192 merged samples, versus 192 evaluations for uniform sampling; the second network also increases memory use.

4.3. Volume Rendering with Signed Distance Functions: VolSDF (10 bonus points)

A neural signed distance function (SDF) defines the surface through VolSDF density, with alpha=80 and beta=0.01 selected by validation PSNR among the seven tested settings.

Alpha scales density, making the interior more opaque as it increases, while beta controls the width of the density transition around the surface; the selected small beta favors sharper boundaries, and alpha=80 gave the best validation result.

Alpha Beta Validation PSNR (dB, 4 views) Test PSNR (dB, 200 views) Test SSIM
10 0.05 21.93 21.69 0.7787
10 0.1 21.43 21.15 0.7583
40 0.05 22.71 22.62 0.8104
40 0.02 22.91 22.83 0.8219
120 0.5 13.67 12.62 0.3824
120 0.01 22.82 22.81 0.8291
80 0.01 22.97 22.96 0.8368
Question Interpretation
How does beta bias the learned SDF? High beta spreads density across a wider region around the zero level set, allowing blurred boundaries and loss of fine geometric detail; low beta concentrates the transition near the zero level set, favoring sharper surfaces.
Is training easier with high or low beta, and why? A moderately high beta is generally easier to train because more samples along each ray receive useful gradients; low beta confines these gradients to a narrow region that coarse sampling can miss.
Which is more likely to recover an accurate surface, and why? Low beta generally favors more accurate geometry because the rendered density boundary lies closer to the SDF zero level set, reducing surface blurring; this requires sufficient sampling and successful optimization, so smaller beta is not always better in practice.

B. 3D Gaussian Splatting (60 points)

5. 3D Gaussian Rasterization (35 points)

Covariance projection uses J R^T Sigma R J^T because PyTorch3D transforms row-vector points as x_view = x_world R + T, so the corresponding column-vector rotation is R^T.

6. Training 3D Gaussian Representations (15 points)

I trained isotropic Gaussians for 1,600 iterations with L1 loss and constant learning rates of 0.0005 for means, 0.005 for scales, 0.01 for colors, and 0.05 for opacities; rotations remain fixed.

Scene PSNR (dB) SSIM Foreground PSNR (dB)
Truck 30.09 0.9450 21.01
Held-out object details: matching crops of prediction and ground truth
Held-out object details: matching crops of prediction and ground truth

7. Extensions (choose at least one; more than one is extra credit)

7.1. Rendering Using Spherical Harmonics (10 points)

Degree-3 spherical harmonics (SH) model view-dependent color, compared at matching viewpoints with the constant-color baseline using only the direct current (DC) term; SH shows more noticeable detail in the chair's material.

7.2. Training On a Harder Scene (10 points)

The materials baseline reuses the Task 6.2 optimizer and full-image L1 training with random initialization, while the improved model uses broader initialization, anisotropic covariance, a combined loss, sampled patches, adaptive splitting and pruning, and longer training.

The baseline remains diffuse during training, while the improved run gradually resolves separate objects and clearer boundaries, though fine reflections remain imperfect.

Held-out PSNR increases from 17.75 to 21.34 dB (+3.59 dB), and SSIM from 0.6357 to 0.8077 (+0.1720); these gains reflect the combined changes and longer training, rather than an isolated test of any one modification.

Held-out image quality
Method PSNR (dB) SSIM Foreground PSNR (dB)
baseline 17.75 0.6357 12.62
improved 21.34 0.8077 16.30
Foreground pixels: ground truth (GT) max RGB > 0.05.
Training and evaluation
Method Iterations Gaussians Test views
baseline 1600 10000 200
improved 8000 18197 200
Training configurations
Setting Baseline Improved
Image resolution 128 x 128 128 x 128
Training supervision Full image (16,384 pixels per step) 2,048 sampled pixels per step in 8 x 8 patches
Covariance Isotropic Anisotropic with learned rotations
Initial positions Zero-mean Gaussian, standard deviation 0.2 Zero-mean Gaussian, standard deviation 0.8
Initial scale uniform random (0, 0.01) root mean squared distance to 3 nearest points
Initial opacity 0.999665 0.1
Initial colors uniform random [0, 1) gray 0.5
Loss 1 L1 + 0 DSSIM 0.8 L1 + 0.2 DSSIM + 0.1 MSE
Final learning-rate factor 1.0 0.1
Adaptive density Disabled Split high-gradient Gaussians along their longest axis; prune low-opacity Gaussians
Mean learning rate 0.0005 0.002
Scale learning rate 0.005 0.005
Color learning rate 0.01 0.01
Opacity learning rate 0.05 0.03
Rotation learning rate Not optimized 0.001
Baseline training progress: predictions above ground truth
Baseline training progress: predictions above ground truth
Improved training progress: predictions above ground truth
Improved training progress: predictions above ground truth
Baseline final views: prediction and ground truth
Baseline final views: prediction and ground truth
Improved final views: prediction and ground truth
Improved final views: prediction and ground truth
Improved materials, held-out view 0: prediction and ground truth
Improved materials, held-out view 0: prediction and ground truth
Improved materials, held-out view 99: prediction and ground truth
Improved materials, held-out view 99: prediction and ground truth
Improved materials, held-out view 199: prediction and ground truth
Improved materials, held-out view 199: prediction and ground truth