Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Tianle Lyu · Andrew ID: tianlel
16-825 Learning for 3D Vision

Setup

Environment Setup

p1

Data

p1

A. Neural Volume Rendering (60 points)

1. Differentiable Volume Rendering

1.1. Familiarize yourself with the code structure

1.2. Outline of tasks

1.3. Ray sampling (5 points)

1.4. Point sampling (5 points)

1.5. Volume rendering (20 points)

2. Optimizing a basic implicit volume

2.1. Random ray sampling (5 points)

2.2. Loss and training (5 points)

BEFORE
AFTER
After optimization, the learned box center is (0.25, 0.25, 0.00), and the learned side lengths are (2.01, 1.50, 1.50). I optimized the box parameters using mean squared error between the rendered RGB values and the ground-truth RGB values. The optimized renderings more closely match the ground-truth box geometry across the four training views.

2.3. Visualization

The spiral rendering of the optimized volume is shown above. The recovered box remains geometrically consistent across viewpoints.

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

Implementation

I implemented the NeRF MLP in implicit.py and used mean squared error for RGB supervision in train_nerf.

Visualization

I implemented a Neural Radiance Field as an MLP that maps 3D sample positions to volume density and RGB color. I used positional encoding for the 3D coordinates, ReLU for density, and Sigmoid for RGB prediction. I trained the model for 250 epochs using mean squared error between the rendered and ground-truth RGB values. The final training loss was approximately 0.0036, and the resulting spiral rendering shows consistent reconstruction of the Lego bulldozer across viewpoints.

4. NeRF Extras (Choose one! More than one is extra credit)

4.1 View Dependence (10 bonus points)

Model Design

I extended NeRF to predict view-dependent color while keeping density dependent only on spatial position. The network takes a sampled 3D position (𝐱) and a normalized ray direction (𝐝), and outputs density (σ(𝐱)) and RGB color (𝐜(𝐱, 𝐝)).

The position is encoded using six harmonic frequencies and passed through eight fully connected hidden layers, each with 256 neurons and ReLU activation. A skip connection concatenates the position encoding with the input to the fifth hidden layer. The resulting features feed two branches: a linear density head followed by ReLU to ensure nonnegative density, and a linear feature head for color prediction.

The normalized direction is encoded using four harmonic frequencies and concatenated with the position features. A small color network, consisting of a 128-neuron ReLU hidden layer and a three-channel sigmoid output layer, predicts RGB values in [0, 1]. Direction information enters only this color branch, allowing appearance to vary with viewpoint without directly changing the predicted density.

For comparison, I used a position-only baseline with the same architecture and parameter count, replacing the direction embedding with zeros. This isolates the effect of direction conditioning. Both models use the same ray sampling and volume rendering pipeline and are trained with RGB mean squared error.

Implementation. Both models were trained on the materials dataset at 128×128 resolution for 300 epochs, using Adam with an initial learning rate of 10−4. Each iteration sampled 1,024 rays, with 128 points per ray between distances 1 and 7. The models used the same architecture, data splits, and random seed, differing only in whether direction information was provided to the color branch. We compared their validation MSE, PSNR, and rendered GIFs at epoch 300.

Results

Position-only Baseline
Epoch 300
View-dependent NeRF
Epoch 300
Model Validation MSE ↓ Validation PSNR ↑
Position-only Baseline 0.004079 23.895 dB
View-dependent NeRF 0.003197 24.953 dB

Both models use their best-validation checkpoints (best.pt).

p1

Discussion and Conclusion. Adding viewing direction improved performance on the materials scene. At epoch 300, the view-dependent model achieved a validation PSNR of 24.953 dB, compared with 23.895 dB for the baseline, and reduced validation MSE by approximately 21.6%. This indicates that direction conditioning benefited held-out viewpoints rather than only fitting the training images.

View dependence helps represent changing highlights and reflections, but an overly expressive direction branch may memorize training views and generalize poorly. Our design limits direction conditioning to a small color branch while keeping density dependent only on position.

Overall, the view-dependent model performed better under the tested settings. However, sharp highlights and complex reflections remained blurred, showing that direction conditioning alone does not fully recover detailed material appearance.

4.2 Coarse/Fine Sampling (10 bonus points)

Model Design

I implemented hierarchical sampling using two independently parameterized NeRF networks. The coarse network first evaluates 64 stratified samples along each ray. Its volume rendering weights define a probability distribution used to draw 64 additional samples through inverse-CDF sampling. The original and additional samples are merged and sorted, and the fine network evaluates all 128 points to produce the final color.

The sampling weights are detached when generating new sample locations. Both networks are trained using RGB supervision: MSE(Ccoarse, CGT) + MSE(Cfine, CGT).

Experimental Setup

I compared a single-network uniform sampling baseline with the coarse/fine model on the materials dataset. Both were trained at 128×128 resolution for 300 epochs, using Adam with an initial learning rate of 10−4 and 1,024 rays per iteration. Both used the same view-dependent network architecture per network, data splits, and random seed.

The baseline evaluated 128 samples per ray. The hierarchical model evaluated 64 coarse samples and 128 merged samples with the fine network, totaling 192 network evaluations per ray. Both methods used stratified perturbation during training and deterministic sampling during evaluation.

Experiments ran sequentially on the same otherwise-idle NVIDIA A100 40GB GPU. Rendering time was measured on the same validation camera after warm-up, using the median of five measurements.

Results

Method Validation MSE ↓ Validation PSNR ↑ Training time ↓ Rendering time per image ↓
Uniform 0.003253 24.878 dB 61.32 min 237.15 ms
Coarse/Fine 0.003102 25.083 dB 115.19 min 340.46 ms
Uniform
Epoch 300
Coarse/Fine
Epoch 300

Both models use their best-validation checkpoints (best.pt).

p1

Conclusion

Hierarchical sampling concentrates additional samples in regions that contribute strongly to the rendered image. In this experiment, it provided a modest quality improvement, but training took approximately 1.88× as long and rendering time increased by 43.6%.

The hierarchical model uses both additional network evaluations and a second independently trained network, so the comparison measures the complete coarse/fine strategy rather than sampling alone. Under these settings, uniform sampling offered a better speed–quality trade-off, while coarse/fine sampling achieved slightly higher reconstruction quality at greater computational cost.

4.3 VolSDF (10 bonus points)

Implementation. I implemented separate MLPs for signed distance and color prediction using positional encoding. The distance network has six linear layers with hidden width 128, while the color network has two linear layers with hidden width 128. The SDF output is unconstrained, allowing negative values inside the object and positive values outside. The color output uses a sigmoid activation. For the distance network, I used Softplus hidden activations and geometric initialization to start from an approximately spherical SDF, initially suppressing the weights associated with the harmonic input features.

Experimental settings. I trained on the Lego bulldozer dataset at 128×128 resolution for 250 epochs per experiment. I compared β ∈ {0.01, 0.02, 0.05, 0.10, 0.20}, keeping α = 10, the Eikonal weight at 0.02, and the network architecture fixed. Each training iteration sampled 1,024 rays, with 128 points per ray between distances 2 and 6. The initial learning rate was 5×10−4.

Best result: β=0.01, 250 epochs

Selected result. I selected β = 0.01 as the best overall setting based on the rendered colors and extracted geometry. It produced a relatively clean appearance while retaining a more complete base and recognizable bulldozer structure. The reconstruction is not perfect: small holes and rough surface details remain. These conclusions are based on qualitative inspection of the rotating renderings.

Interpretation of α and β. The parameter α sets the density scale: increasing it increases light attenuation inside the object. The parameter β controls the width of the density transition around the surface. Larger values create a broader transition, while smaller values make the transition sharper.

COLOR
GEOMETRY β=0.02 β=0.05 β=0.10 β=0.20
1. How do high and low β bias the learned SDF?

A larger β allows appreciable density farther outside the zero level set. Consequently, the model can explain image appearance using a wider transition region, and the learned surface may shift or lose geometric detail. A smaller β ties the density boundary more closely to the SDF zero level set. This does not imply that changing β always causes a particular direction of expansion or contraction.

2. Is training easier with low or high β?

A moderately larger β generally provides useful density gradients over a wider spatial region, which can make optimization easier. With a small β, the transition is narrow and requires sufficiently dense sampling near the surface. Our uniform sample spacing is approximately 4/127 ≈ 0.0315, so the β=0.01 experiment is particularly sensitive to sampling. However, the final renderings alone do not establish which setting converged faster.

3. Is an accurate surface more likely with high or low β?

With adequate sampling and successful optimization, a smaller β can support more precise surface localization because the density transition is sharper. In these experiments, β=0.01 gave the best qualitative balance, whereas β=0.20 produced substantial geometric artifacts. The results were not strictly monotonic: β=0.10 had cleaner color renderings than β=0.05, but its geometry still contained visible holes. This also shows why both color and geometry should be evaluated.

B. 3D Gaussian Splatting (60 points)

5. 3D Gaussian Rasterization (35 points)

5.1 Project 3D Gaussians to Obtain 2D Gaussians

Already completed in model.py.

5.2 Evaluate 2D Gaussians

Already completed in model.py.

p1 All test cases have passed.

5.3 Filter and Sort Gaussians

Already completed in model.py.

5.4 Compute Alphas and Transmittance

Already completed in model.py.

5.5 Perform Splatting

p1

6. Training 3D Gaussian Representations (15 points)

6.1 Setting Up Parameters and Optimizer

Already completed in train.py.

6.2 Perform Forward Pass and Compute Loss

I trained the isotropic Gaussian representation of the toy truck for 4,000 iterations, using the Adam optimizer and an L1 loss between the rendered images and the ground-truth images.

The learning rates were:

Parameter Learning rate
Means0.00016
Scales0.003
Colours0.0025
Opacities0.025

Evaluation on the held-out test views produced a mean PSNR of 31.031 dBand a mean SSIM of 0.951. This configuration achieved higher scores than my initial 1,000-iteration experiment, which obtained 29.146 dB PSNR and 0.937 SSIM Both the learning rates and the iteration count changed between the experiments.

The following GIFs show the training progress and the final rendered views. In the training progress GIF, the top row shows the predicted renderings, while the bottom row shows the ground-truth images.

p1 p1 p1

7. Extensions (Choose at least one! More than one is extra credit)

7.1 Rendering Using Spherical Harmonics (10 points)

7.1 (this question)
5.5 (older question)
7.1 (this question)
5.5 (older question)

For column 1, the rendering with spherical harmonics (7.1) shows darker green upholstery, especially on the seat, than the DC-only rendering (5.5). The chair’s overall shape remains unchanged.

For column 2, the backrest and seat appear darker in 7.1, making the gold patterns stand out more against the green fabric. The DC-only rendering appears brighter and more uniform.

For column 3, the two renderings look very similar from the underside. Only subtle differences in shading are visible around the seat base and frame.

For column 4, the front-facing backrest is noticeably darker in 7.1, increasing the contrast between the green fabric and the gold patterns. The silhouette remains consistent between the two renderings.

7.2 Training On a Harder Scene (10 points)

Experimental Results

ID Gaussian Type Loss Function Iterations Learning Rate Schedule PSNR ↑ SSIM ↑
A Isotropic L1 1,000 Fixed 17.585 0.645
B: Official Baseline Isotropic L1 4,000 Fixed 18.517 0.687
C Isotropic L1 10,000 Decay to 10% of the initial value 18.504 0.686
D: Highest PSNR Isotropic L1 10,000 Fixed 18.680 0.704
E Isotropic L1 + SSIM 6,000 Fixed 18.355 0.704
F Anisotropic L1 + SSIM 6,000 Fixed 18.497 0.709
G: Highest SSIM Anisotropic L1 + SSIM 10,000 Fixed 18.575 0.715

Baseline and Final Configuration

Setting Baseline Final Method
Initialization Random, 10,000 Gaussians Random, 10,000 Gaussians
Gaussian Type Isotropic Anisotropic
Loss Function L1 0.8 × L1 + 0.2 × (1 − SSIM)
Optimizer Adam Adam
Position / Scale Learning Rate 0.00016 / 0.003 0.00016 / 0.003
Color / Opacity Learning Rate 0.0025 / 0.025 0.0025 / 0.025
Rotation Learning Rate N/A (rotation not optimized) 0.001
Training Iterations 4,000 10,000
Learning Rate Decay None None
Mean PSNR (dB) 18.517 18.575
Mean SSIM 0.687 0.715

Final Model Design

The model represents the materials scene using 10,000 randomly initialized anisotropic 3D Gaussians. Each Gaussian has a learnable position, three independent scale parameters, a rotation quaternion, an RGB color, and an opacity. The independent scales and rotation allow each Gaussian to vary its shape and orientation.

For each camera view, the renderer filters out Gaussians with negative depth, sorts the remaining Gaussians from front to back, and projects them onto the image plane. It computes per-pixel alpha values and transmittance, then blends the Gaussian colors to produce the predicted image.

The model is trained with the combined objective: L = 0.8 × L1(Ipred, Igt) + 0.2 × (1 − SSIM(Ipred, Igt)). Both terms are differentiable, allowing gradients to propagate through the renderer to the Gaussian parameters. Training uses Adam for 10,000 iterations with fixed learning rates: 0.00016 for positions, 0.003 for scales, 0.0025 for colors, 0.025 for opacities, and 0.001 for rotations. Scales use an exponential activation, opacities use a sigmoid activation, and rotation quaternions are normalized before rendering. The number of Gaussians remains fixed throughout training, with no adaptive densification or pruning.

Visualization

Baseline
Final Method
Baseline Baseline rendered result
Final Method Final method rendered result

Compared with the baseline using the Q6.2 training setup, my final method increases training from 4,000 to 10,000 iterations, replaces isotropic Gaussians with anisotropic Gaussians, and uses a combined loss of 0.8 × L1 + 0.2 × (1 − SSIM). The anisotropic representation also optimizes quaternion rotations with a learning rate of 0.001. Both methods use random initialization with 10,000 Gaussians and fixed learning rates.

The final method improves PSNR from 18.517 to 18.575 dB and SSIM from 0.687 to 0.715. I selected this method because it achieves the highest SSIM among my experiments. However, the isotropic L1 method trained for 10,000 iterations achieves the highest PSNR, at 18.680 dB, so no single configuration performs best on both metrics.

The experiments show that longer training improves the fixed-learning-rate L1 results. At 6,000 iterations with the same combined loss, anisotropic Gaussians improve both PSNR and SSIM over isotropic Gaussians. In contrast, decaying all learning rates to 10% of their initial values does not improve the results in this experiment.

The GIFs show the progression from random initialization to a recognizable reconstruction. Both methods still exhibit blurred objects and visible artifacts, so the improvement remains limited. Since the final method changes the training duration, loss function, and Gaussian type together, its improvement over the baseline cannot be attributed to any one modification alone.