Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

16-825: Learning for 3D Vision · Shohei Nagai (Andrew ID: shoheina)

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

I implement get_pixels_from_image to map pixel coordinates to NDC, and get_rays_from_pixels to unproject them with camera.unproject_points into rays from the camera centre. Both visualizations match the TA images.

Sampled pixel grid (xy)Visualized camera rays
Figure 1: XY grid visualization (left) and ray visualization (right).

1.4 Point sampling

I implement StratifiedRaysampler to place n_pts_per_ray depths uniformly between min_depth and max_depth and sample points at origin + depth × direction.

Sampled points
Figure 2: Sampled points.

1.5 Volume rendering

I implement _compute_weights and _aggregate as the weighted sum along each ray. The same weights render the color and the depth.

Rendered imageDepth map
Figure 3: Rendered image (left) and depth map normalized by its maximum (right).

2. Optimizing a Basic Implicit Volume

2.1 Random ray sampling

I implement get_random_pixels_from_image to pick n_pixels random pixels from the full grid, so each step renders only a batch of rays.

2.2 Loss and training

I define the loss as the MSE between the rendered colors and rgb_gt. After training, the box is:

Box center: (0.25, 0.25, 0.00)
Box side lengths: (2.01, 1.50, 1.50)

2.3 Visualization

The spiral rendering of the optimized box matches the TA GIF in shape, size and position.

Optimized volume
Figure 4: Optimized volume.

3. Optimizing a Neural Radiance Field (NeRF)

Network. Positional encoding of xyz (6 frequencies) → 6-layer MLP (128 units, skip connection at layer 3) → ReLU density head and sigmoid color head. Loss. MSE on 1024 random rays per step.

NeRF rendering
Figure 5: NeRF trained on the lego scene (nerf_lego.yaml, 128x128, 250 epochs).

The rendering matches the TA reference: the shape of the bulldozer, the yellow body, the dark treads and the textured base are all reconstructed, and the test PSNR is 24.50 dB.

4. NeRF Extras

4.1 View Dependence

Network. Following the NeRF paper, I feed the encoded viewing direction only into the color head, after the density head. The density depends only on the position, so the network cannot overfit the training views with view-dependent geometry.

L is the number of frequencies in the harmonic encoding of the viewing direction d: γ(d) = (d, sin(2^0 d), cos(2^0 d), …, sin(2^(L−1) d), cos(2^(L−1) d)), which has 3 + 6L dimensions. A larger L lets the color change faster as the viewpoint moves, so it can represent sharper highlights but can also fit view-specific details of the training images. The baseline without view dependence ignores the direction entirely.

I trained on the materials scene (nerf_materials.yaml, 128x128, 1000 epochs, learning rate 5e-4) without view dependence and with L = 2, 4 and 8. The highlight PSNR is computed only on the brightest pixels (any channel above 0.8), which are the specular highlights.

ModelTrain PSNRTest PSNRTrain-test gapHighlight PSNR (train / test)
No view dependence25.8824.261.6111.60 / 11.63
L=228.5126.272.2415.37 / 13.94
L=428.6926.312.3816.04 / 14.10
L=828.8826.232.6515.97 / 13.98
View dependence comparison
Figure 6: Test view rendered by each model, next to the ground truth.
Materials, L=2, 128x128Materials, L=2, 400x400
Figure 7: View-dependent NeRF (L=2) on the materials scene, trained with low-resolution 128x128 (left) and high-resolution 400x400 (right).

Trade-off between view dependence and generalization:

In summary, a small L (2–4) gives most of the benefit, and a larger L mainly widens the train-test gap. Even so, the mirror-like reflections stay blurrier than the ground truth (Figure 6). The reflected image changes very sharply with both the viewing direction and the position, which this small color head cannot represent.

4.2 Coarse/Fine Sampling

Sampling. A coarse network is evaluated at 64 stratified samples. I draw 128 more samples from its rendering weights by inverse-transform sampling (sample_pdf), and a fine network is evaluated at all 192. Loss. MSE on both renderings. The figure compares uniform 128 samples (Q3), uniform 192, and coarse 64 + fine 128.

Sampling SchemeTest PSNRTrain Step (ms)Render 128px (s)# Params
uniform_128 (Q3)24.4978.10.031117,764
uniform_19224.75412.950.049117,764
coarse_64 + fine_12824.93616.260.061235,528
Sampling comparison
Figure 8: Sampling comparison.

Coarse/fine sampling is worth the cost when quality matters. When speed matters, adding uniform samples recovers part of the gain more cheaply.

4.3 VolSDF

I implement VolSDF following items described in the instructions:

I sweep β from 0.01 to 0.5 (α = 10). Depth gap is the distance between the rendered depth and the SDF zero crossing. Empty-ray opacity is the opacity of rays that miss the surface (haze).

βTest PSNRDepth gapEmpty-ray opacity
0.0121.210.0990.003
0.0520.960.0890.043
0.220.130.2160.465
0.518.700.3950.679
Beta sweep
Figure 9: color (top) and mesh (bottom) for each β.
  1. How does β bias the SDF? High β spreads density away from the zero level set, so the SDF learns a bloated shape that loses thin parts (Figure 9, β ≥ 0.2). Low β makes the rendered surface coincide with the zero level set, so there is little bias.
  2. Which is easier to train? High β. Many samples along each ray get gradients, even when the surface is far from where it should be. With low β, only samples within ~β of the surface get gradients, and β = 0.01 is smaller than the sample spacing (0.03).
  3. Which gives a more accurate surface? Low β. Rendering is determined by the zero level set itself, so fitting the images places it on the true surface. This gives the sharpest mesh and no haze.

Settings. α = 10, β = 0.01, eikonal weight 0.02, 250 epochs. The key change is a geometric initialization (from IDR/VolSDF) that starts the network as the SDF of a sphere. With random initialization, unconstrained regions keep random zero crossings, which become spurious surfaces in the mesh (Figure 10). I kept the default α = 10: a ray is about 95% opaque within 0.3 units (about 10 samples) past the surface, so the color comes from a thin shell at the surface. I also kept the default eikonal weight of 0.02, which keeps ‖∇d‖ close to 1 so that d stays a true distance and β keeps its meaning, without overpowering the color loss.

Random initialization
Figure 10: The same sweep with random initialization.
VolSDF geometryVolSDF color
Figure 11: Final result (β = 0.01): mesh (left) and color (right).
B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

I implement the rasterizer in model.py and render.py, vectorized over all Gaussians:

The GIF shows the color, depth and mask of the pre-trained chair rendered by render.py.

Gaussian Rasterization
Figure 12: Gaussian rasterization of the pre-trained chair.

6. Training 3D Gaussian Representations

I train 10,000 isotropic Gaussians on the toy truck, initialized from the provided point cloud.

6.1 Parameters and optimizer. make_trainable makes the means, scales, opacities and colors trainable, and setup_optimizer uses Adam with a learning rate per parameter:

ParameterLearning rate
Opacities0.005
Scales0.01
Colors0.005
Means0.01

The starter rate of 0.05 was too high, so I lowered all of them, most for the colors and opacities.

6.2 Forward pass and loss. Each iteration renders one training view with scene.render and computes the MSE against the ground truth.

IterationsPSNRSSIM
100028.9030.926
Gaussian Training Progress
Figure 13: Training progress. Top: renders of 4 training views. Bottom: ground truth.

At iteration 0 the Gaussians already form the truck, since they start from the point cloud, but their colors are random. The colors and silhouette match the ground truth after about 200-300 iterations, and the rest of training sharpens details such as the wheels and windows.

Gaussian Training Final Renders
Figure 14: Final renders of all training views after 1000 iterations.

7. Extensions

7.1 Rendering Using Spherical Harmonics

I implement colors_from_spherical_harmonics, which evaluates spherical harmonics up to degree 3 along the direction from the camera to each Gaussian. The baseline (Q5.5) uses only the degree-0 term, which is the same from every direction.

Spherical Harmonics Rendering
Figure 15: Rendering with spherical harmonics (Q7.1).
Baseline rendering
Figure 16: Baseline rendering without view dependence (Q5.5).

Side-by-side comparison of the same frames, with the baseline (top) and with spherical harmonics (bottom):

Spherical harmonics comparison
Figure 17: Frames 012, 015 and 017 rendered with the baseline (Q5.5, top) and with spherical harmonics (Q7.1, bottom).

7.2 Training On a Harder Scene

Both methods start from 10,000 random Gaussians and train for 5000 iterations. The table below shows the parameters used for each training configuration.

Baseline (Q6 setup)Improved
Learning ratesopacities 0.005, scales 0.01, colors 0.005, means 0.01opacities 0.05, scales 0.005, colors 0.01, means 0.005 → 0.00005, quaternions 0.001
Iterations50005000
PSNR21.43024.381
SSIM0.6260.891
Training progress comparison
Figure 18: Training progress on 4 training views for ground truth (top), baseline (middle) and improved method (bottom).
Baseline final rendersImproved final renders
Figure 19: Final renders for the baseline (left) and the improved method (right).

The baseline Gaussians start as a small ball and stay speckled and noisy. The improved Gaussians cover the scene from the start, place the objects within about 500 iterations, and end much sharper.

Modifications compared with Q6:

  1. Initialization: means uniform over the whole scene (a cube of half size 1.5) instead of a small ball, scales from the distance to the nearest neighbors, and a low initial opacity (0.1).
  2. Anisotropic Gaussians: a scale per axis plus a rotation, to fit edges and surfaces.
  3. L1 + D-SSIM loss (weight 0.2 on D-SSIM), as in the original paper.
  4. Learning-rate schedule: the means' learning rate decays exponentially by 100× over training.
  5. Simplified adaptive density control: every 200 iterations from 500 to 3000, the 5% of Gaussians with the largest mean gradient are cloned (if small) or split (if large). Nearly transparent or very large Gaussians are pruned.