Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)

A. Neural Volume Rendering

1.3 Ray Sampling (5 points)

Left: xy_grid. Right: rays.

Pixel grid visualization Ray direction visualization

1.4 Point Sampling (5 points)

Sample points along rays from the first camera

1.5 Volume Rendering (20 points)

Left: color. Right: depth, divided by its maximum value.

Rendered box color from surround views Normalized depth of the box

2.2 Loss and Training (5 points)

Box center: (0.25, 0.25, 0.00)

Box side lengths: (2.00, 1.50, 1.50)

2.3 Results

Optimized box from surround views

3. Optimize a Neural Radiance Field (20 points)

MLP: 6 layers of 128 with ReLU, position input added again at layer 3, 10 harmonic functions for position. No view dependence. 250 epochs, lr 5e-4, 128 x 128 images.

Mean validation PSNR: 23.82 dB (100 views).

NeRF on lego from surround views

4.1 View Dependence (10 bonus points)

Materials scene, 400 x 400. Both runs use the same settings: 6 layers of 256, 10 harmonic functions for position, 400 epochs, lr 1e-4. In the view-dependent run, density uses only the position. The color head takes the position features and the view direction (2 harmonic functions), with one hidden layer of 128.

ModelValidation PSNR (dB, 100 views)
No view dependence23.94
View dependence25.18

Left: no view dependence. Right: view dependence.

Materials scene without view dependence on the left and with view dependence on the right

Train and validation curves. Top: no view dependence. Bottom: view dependence.

Train and validation loss and PSNR without view dependence
Train and validation loss and PSNR with view dependence

View dependence gives higher PSNR on both train and validation views. But I think the two look very similar, and the effect of view dependence is not obvious. So I tried the parameters from the paper, with a bigger and deeper network, and then you can see the difference (below).

In both runs, the gap between train and validation PSNR stays below 0.5 dB, so I don't see clear overfitting at 400 epochs. I think this comes from the network design in the NeRF paper: the position goes through a shared backbone, and density comes only from it, so the shape is the same from all views. The view direction is only added to a small color head on top of the backbone features. So the backbone has to learn one shape for all views, and the direction only changes the color on it.

According to NeRF++ (Zhang et al. 2020), the risk of view dependence is the shape-radiance ambiguity: a wrong shape can still match all training images if its color can change freely with the view, but new views will be wrong. In NeRF, the direction only has a few frequencies (2 in my runs, 4 in the paper) and a small head, so the color can only change a little with the view. This is not enough to hide a wrong shape, so the network has to learn the correct shape.

Best Result: NeRF Paper Architecture

Network from the NeRF paper: 8 ReLU layers of 256, position input added again at layer 5, a linear feature layer, and a color head of 128. 10 harmonic functions for position and 4 for direction. 128 samples per ray (no coarse/fine sampling), 1024 rays per step, lr 5e-4, 400 epochs.

With PyTorch default initialization, this deep network collapsed to an empty scene: density was 0 everywhere and got no gradient. These changes fixed it: Kaiming initialization, density bias 0.1, rays only from the image center (half size) for the first 5 epochs, and a fixed seed.

ModelValidation PSNR (dB, 100 views)
Paper architecture, view dependence27.41
Materials scene with the NeRF paper architecture and view dependence

Two train views and two validation views. Left to right: ground truth, prediction, absolute error x4. Brighter means larger error.

Ground truth, prediction, and absolute error for two train views and two validation views

I find the mirror material shows larger error, which I think can be explained by that our model tends to blur things and may fit diffuse surface appearance better.

Train and validation loss and PSNR for the NeRF paper architecture

The metal highlights are clear, and the mirror sphere shows the environment. The gap between train and validation PSNR is about 1 dB (28.1 and 27.2 on random rays at epoch 400), which is larger than in the smaller runs. A larger network and more direction frequencies fit the train views better, but validation still improves with train, so it does not overfit badly at 400 epochs.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization (35 points)

Rendered pre-trained chair Gaussians

6. Train 3D Gaussian Representations (15 points)

Isotropic Gaussians, L1 loss, Adam, 1000 iterations.

ParameterLearning rate
Opacities0.05
Scales0.005
Colors0.02
Means0.0005

Mean PSNR: 29.54. Mean SSIM: 0.936.

Training progress. Top: prediction. Bottom: reference.

Training progress: predictions above reference images

Final renders of training views.

Final renders of the truck training views

7.1 Render with Spherical Harmonics (10 points)

Without spherical harmonics (only the constant component):

Chair rendered with only the constant color component

With spherical harmonics (degree 3):

Chair rendered with view-dependent spherical harmonic color

Same views side by side. Left: without spherical harmonics. Right: with spherical harmonics.

Without spherical harmonics, each Gaussian has one color for all views. With spherical harmonics, I can very clearly see that when the chair rotates, the silk velvet cushion gets brighter or darker at different angles, and the strength of the reflection also changes.

View 1: chair front without and with spherical harmonics

View 1: in the right image, the center of the cushion has a very clear golden reflection, and there is a shading from left to right. There is no sharp boundary between light and dark like in the left image; it is softer and silkier.

View 10: chair from the side without and with spherical harmonics

View 10: in the right image, the position of the reflection is completely different from view 1, and the contrast between light and dark is much better. In the left image, the colors of all parts are very thin and uniform. It feels like the left image has a soft light from all directions, and the right image has a more real light source.