16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Liza Dahiya

A. Neural Volume Rendering

1. Differentiable Volume Rendering

All Part 1 results use configs/box.yaml: a 256×256 image, 64 samples per ray between depths 0 and 5, and a box SDF (side 1.75) converted to density with α = 1, β = 0.05.

1.3 Ray Sampling (5 pts)

Each pixel center is mapped to normalized device coordinates in [-1, 1], placed on the image plane at depth 1, and unprojected to world space with camera.unproject_points(..., from_ndc=True). Rays start at the camera center and point (normalized) through those image-plane points.

Pixel grid visualization
Pixel grid (xy in [-1, 1] shown as red/green)
Ray direction visualization
Ray directions (|direction| shown as RGB)

1.4 Point Sampling (5 pts)

The stratified sampler places n_pts_per_ray evenly spaced depths between min_depth and max_depth, and each sample point is origin + depth × direction.

Sample points from the first camera
Sample points along every ray of the first camera

1.5 Volume Rendering (20 pts)

For a ray with N samples at depths t1 < … < tN, with density σi and color ci at each sample:

Δti = ti+1 − ti,    ΔtN = 1010
αi = 1 − exp(−σi Δti)
Ti = ∏j=1i−1 (1 − αj) = exp(−∑j=1i−1 σj Δtj),    T1 = 1
wi = Ti αi
C = ∑i=1N wi ci,    D = ∑i=1N wi ti,    Dvis = D / maxpixels D

_compute_weights computes Ti as an exclusive cumulative product of (1 − αj + 10−10) along the ray, and _aggregate takes the weighted sum, used for both color and depth.

Spiral rendering of the box
Spiral rendering
Depth map of the box
Depth (camera 2), normalized by its maximum

2. Optimizing a Basic Implicit Volume

Using configs/train_box.yaml, I optimized the box's center and side lengths (starting from center (0, 0, 0) and sides 1.5) from four ground-truth images with known camera poses. Training used Adam (lr 5e-4) for 1,000 epochs.

2.1 Random Ray Sampling (5 pts)

Each iteration renders 4,096 rays rather than the full 256×256 image. I build the full pixel grid, shuffle it with torch.randperm, and keep the first 4,096 pixels, so each pixel is picked at most once per batch.

2.2 Loss and Training (5 pts)

The loss is the mean squared error between the rendered colors and the ground-truth colors at the sampled pixels. It fell to about 1e-6 by the end of training.

xyz
Box center0.250.250.00
Box side lengths2.011.501.50

2.3 Visualization

Spiral rendering of the optimized box
Spiral rendering of the optimized box
TA reference spiral rendering
TA reference

3. Optimizing a Neural Radiance Field (NeRF) (20 pts)

The NeRF maps a 3D position to density and color. It doesn't use the viewing direction. Settings follow configs/nerf_lego.yaml:

The loss is MSE between rendered and ground-truth pixel colors. I trained for 250 epochs on 128×128 lego images with 1,024 rays per batch and 128 samples per ray between depths 2 and 6. I used Adam with lr 5e-4, decayed by 0.8 every 50 epochs. The mean loss per epoch fell from 0.088 in the first epoch to 0.0022 in the last, and PSNR on the 100 held-out validation views is 26.3 dB.

Part 3 NeRF training loss curve
Training loss, log scale. Most of the drop happens in the first 20 epochs; after that the loss keeps decreasing slowly as the learning rate decays.
NeRF spiral rendering of the lego bulldozer
My NeRF
TA reference NeRF rendering
TA reference

4. NeRF Extras

I did all three extensions. Every model trains for 250 epochs with the same schedule as Part 3. PSNR is the average over the dataset's 100 training views and its 100 held-out validation views. The rendering is compared against the ground truth sampled exactly as during training, since the starter's sample_images_at_xy is offset by up to half a pixel from the raw image grid.

4.1 View Dependence (10 bonus pts)

As in NeRF, density depends only on position, so geometry stays consistent across views. For color, the trunk's features pass through a linear layer and are concatenated with the encoded viewing direction (2 frequencies, 15 dimensions). A 64-unit layer then predicts RGB. I compared it with the position-only model from Part 3 on the materials scene, at both the 128×128 and 400×400 settings from the provided configs.

Materials sceneTrain PSNRVal PSNRTrain − val gap
128×128, position only24.7924.640.15
128×128, view dependent25.8425.650.19
400×400, position only23.9523.820.13
400×400, view dependent24.9924.830.16
Materials, position-only NeRF
400×400, position only
Materials, view-dependent NeRF
400×400, view dependent

The materials spheres are glossy and metallic, so their highlights move as the camera moves. A position-only model can't represent that: it averages the highlights from all training views into each point, which shows up as a speckled, blotchy texture. The view-dependent model renders smooth surfaces with coherent highlights, and it gains about 1 dB on held-out views at both resolutions.

Trade-off. More view dependence lets the network explain each training image with color alone. In the extreme, it can memorize each view's appearance and leave the geometry wrong, which then breaks on new views. The train−val gap grows slightly with view dependence here (0.15 → 0.19 dB and 0.13 → 0.16 dB), but held-out PSNR still improves clearly. Three choices keep this in check, all following NeRF:

Feeding direction in earlier or with more frequencies would fit the training views more closely but generalize worse.

4.2 Coarse/Fine Sampling (10 bonus pts)

Two separate NeRFs. The coarse network is evaluated at Nc stratified samples, jittered randomly within their bins during training. Its weights (detached) define a piecewise-constant PDF along each ray, and inverse transform sampling draws Nf more depths where the weights are large. The fine network is evaluated at all Nc + Nf sorted depths and produces the final color. Both networks are trained with the MSE loss. I compared it with uniform sampling on lego at two budgets. Times were measured alone on one H200 (1,024-ray training step; full 128×128 render).

LegoPoints per ray (coarse + fine net)Val PSNRTrain stepRender
Uniform 646425.255.5 ms10.5 ms
Coarse 32 + fine 3232 + 6426.2610.3 ms15.4 ms
Uniform 128 (Part 3)12826.326.2 ms18.9 ms
Uniform 19219226.488.0 ms28.3 ms
Coarse 64 + fine 12864 + 19226.4915.1 ms38.2 ms
Uniform 64 samples
Uniform 64
Coarse 32 + fine 32
Coarse 32 + fine 32
Coarse 64 + fine 128
Coarse 64 + fine 128

Quality. Coarse/fine sampling helps most when samples are scarce. With 64 points per ray for the fine network, it beats uniform 64 by 1.0 dB and matches uniform 128. It concentrates samples near surfaces instead of in empty space. At 128 or more uniform samples, a 128×128 lego image is already close to saturated, and coarse/fine gives about the same PSNR as uniform 192.

Speed. It costs a second network, a second forward/backward pass, and the sampling step. A training step takes about 2× as long as uniform sampling with the same number of fine-network points, and rendering about 1.5×. It also doubles the parameter count. So it is worth it when samples are the bottleneck (higher resolution, larger depth ranges, thin structures). It isn't worth it when uniform sampling is already dense.

4.3 VolSDF (10 bonus pts)

What α and β do. α is the density inside the object: far inside, σ → α; exactly on the surface, σ = α/2; and outside, it decays to 0. β is how sharply density drops off across the surface. Density goes from about α to about 0 over a band a few β wide around the zero level set. As β → 0, this becomes a step function: a solid object with an infinitely thin surface.

  1. How β biases the SDF. With high β, density is smeared over a thick shell, so rays start accumulating opacity well before reaching the zero level set. To match the images, the learned surface shrinks inward and fine structures erode or break apart. In my β = 0.3 result the shovel arm separates from the body, and the result looks blurry, like a fog. Low β makes the rendered surface coincide with the zero level set, so the SDF is pulled toward the true surface. But each surface only gets gradient from a thin band, so regions the rays don't constrain well can keep floaters.
  2. Easier to train: high β. The wide density band means every sample near the surface gets a non-zero gradient, and distant parts of the SDF still feel the photometric loss. With very low β, a ray only gets useful gradient if a sample happens to land in the thin band around the surface. The loss is much less smooth, and early in training (when the surface is in the wrong place) gradients are weak or noisy.
  3. Accurate surface: low β. Once training succeeds, low β makes the opacity transition happen exactly at the zero level set. The extracted mesh then matches what is rendered, rather than sitting inside a blurry density shell. This is why VolSDF learns β and lets it shrink during training.
β (α = 10)Train PSNRVal PSNR
0.0224.1423.93
0.0524.1923.96
0.123.8523.64
0.322.3322.16
VolSDF geometry, beta 0.02
β = 0.02
VolSDF geometry, beta 0.05
β = 0.05
VolSDF geometry, beta 0.1
β = 0.1
VolSDF geometry, beta 0.3
β = 0.3

Best result (β = 0.05, α = 10). β = 0.05 gave the best held-out PSNR, and β = 0.02 was within 0.03 dB. Both keep the thin arm and shovel attached and the tracks well defined. β = 0.05 is a good middle ground: it is narrow enough that the mesh matches the rendered surface, and wide enough that rays get gradient early in training. Two other choices also helped:

Best VolSDF geometry
Geometry (β = 0.05)
Best VolSDF color rendering
Color (β = 0.05)

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization (35 pts)

A vectorized PyTorch rasterizer, used to render the provided pre-trained chair (283,873 Gaussians) from 32 views at 256×256, splatting 2,048 Gaussians per batch.

Rendered chair: color, depth and silhouette
Pre-trained chair with W = RT (final): color (white background), depth, silhouette

5.1 Note: camera rotation convention (W = RT vs W = R)

The provided unit tests and reference renders use W = R, PyTorch3D's rotation matrix as stored, and the instructors allowed that convention on Piazza. I use W = RT for the final results, because it is the one that matches PyTorch3D's projection. I rendered Part 5 both ways for comparison (python render.py and python render.py --w_equals_r).

W = RT (final)W = R (unit tests' convention)
Unit tests passed2 / 4 (tests 2 and 4 fail)4 / 4
Matches autograd Jacobian of the projectionyes (~1e-7)no (up to 75% off)
Chair rendered with W = R transpose
W = RT (final)
Chair rendered with W = R
W = R

Same frames, W = RT (left) vs W = R (right):

Frame 23: W = R transpose vs W = R
Frame 23
Frame 10: W = R transpose vs W = R
Frame 10

With W = R, every splat's footprint is rotated by the wrong rotation. Elongated Gaussians get stretched in directions that don't follow the surface. The underside of the seat in frame 23 turns into a brushed, streaky texture, and spikes stick out of every edge: legs, armrests and frame. The same fringe is visible in the provided reference images. With W = RT, the seat underside is smooth and the silhouette is sharp. Both renders use the same Gaussians, so the difference comes entirely from the projection.

I also re-ran Parts 6 and 7 with W = R (the rest of the page shows the W = RT results):

PSNR / SSIMW = RT (final)W = R
Q6 truck (isotropic)29.36 / 0.93929.42 / 0.938
Q7.2 baseline (isotropic)18.48 / 0.68418.67 / 0.683
Q7.2 improved (anisotropic)24.42 / 0.89124.01 / 0.875

For isotropic Gaussians, Σ = s²I, so WΣWT = s²I for any rotation W, and the convention has no effect. The small Q6 and Q7.2 baseline differences are just run-to-run randomness. Only the anisotropic model depends on the convention. It does slightly better with W = RT (+0.4 dB, +0.016 SSIM). The gap is small because training is self-consistent either way: with W = R, the learned orientations partly adapt to the wrong projection.

6. Training 3D Gaussian Representations (15 pts)

The model uses 10,000 isotropic Gaussians, with means initialized from the provided truck point cloud. They are trained with the Part 5 rasterizer on the 197 training views, using an L1 loss between the rendered image (black background) and the ground truth. The optimizer is Adam with a separate learning rate for each parameter. Positions get the smallest rate: a small shift in a mean moves a Gaussian across pixels, while opacity and color can change quickly without destabilizing the scene.

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.02
Means0.0005

I trained for 1,000 iterations (one view per iteration). The L1 loss fell from 0.041 to 0.005. On the 5 held-out test views:

Mean PSNRMean SSIM
29.360.939
Training progress: renders (top) vs ground truth (bottom)
Training progress: current renders (top) vs ground truth (bottom), every 20 iterations
Final renders of all training views
Final renders of the training views

7. Extensions

I did both extensions.

7.1 Rendering Using Spherical Harmonics (10 pts)

The pre-trained chair stores 16 spherical-harmonic coefficients per colour channel (degrees 0–3). Question 5 used only the degree-0 (DC) term, which is the same from every direction. For 7.1, each Gaussian's colour is computed from all 16 coefficients:

Chair rendered with DC colour only
Question 5.5: DC colour only
Chair rendered with spherical harmonics
Question 7.1: full spherical harmonics

Side by side comparisons of the same frame, DC only (left) vs full SH (right). These are the three frames where the two renders differ most.

Frame 14: DC only vs SH
Frame 14

The upholstery is a darker, more saturated green with SH, and the gold floral pattern on the backrest stands out clearly. With DC only, the same fabric looks flat and washed out. The DC term is effectively the colour averaged over all viewing directions, so the sheen the fabric shows at some angles is spread evenly over every view and lightens it. The higher-order terms put that sheen back only where it appears from this viewpoint.

Frame 1: DC only vs SH
Frame 1

The DC-only render has a large pale-green patch across the lower middle of the backrest and lighter stripes between the motifs. That is sheen averaged in from other viewpoints. With SH, the same area is a deep, even green, and the gold motifs, especially the central one, become bright and metallic. Gold thread is specular, so it catches light only from some directions, and only the higher-order SH terms can represent that.

Frame 31: DC only vs SH
Frame 31

Seen from the front, the DC-only render again shows washed-out highlights in the middle of the backrest and on the seat cushion. With SH, those disappear, the fabric is darker and more saturated, and the central gold motif reads as shiny embroidery rather than a flat print. Geometry, silhouettes and depth maps are identical in both renders, since SH only changes colour. The difference is purely view-dependent appearance, and it matches the darker, more contrasted look of the TA's with-SH reference.

7.2 Training On a Harder Scene (10 pts)

Both runs train 10,000 Gaussians from a random initialization on the 100 materials training views at 128×128 and are evaluated on the 100 validation views. train_harder_scene.py --method baseline and --method improved reproduce them.

Baseline (question 6 setup)

Isotropic Gaussians and init_type="random". The learning rates and L1 loss are the same as question 6 (opacities 0.05, scales 0.005, colours 0.02, means 0.0005), trained for 1,000 iterations.

Mean PSNRMean SSIM
Baseline18.480.684
Baseline training progress
Baseline: training progress (renders top, ground truth bottom)
Baseline final renders
Baseline: final renders of the training views

The random initialization puts every Gaussian in a small ball (σ = 0.2) at the origin, but the spheres in this scene spread over roughly ±1.1 units. Starting there with a small learning rate, the Gaussians can't travel far enough in 1,000 steps. The central spheres are reconstructed as blurry clumps, and the outer ones appear only as faint blobs or not at all.

Improved method

Same number of Gaussians (10,000), the same random-init entry point (init_type="random") and the same data, trained for 3,000 iterations.

IterationsMean PSNRMean SSIM
Baseline (question 6 setup)1,00018.480.684
Improved3,00024.420.891
ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.02
Means0.002, decayed exponentially to 0.00002 by the last iteration
Quaternions0.002
Improved training progress
Improved: training progress (renders top, ground truth bottom)
Improved final renders
Improved: final renders of the training views

The improved model recovers all 13 spheres in the right places and colours. The baseline only produces blurry clumps near the centre. What's still missing is the sharp view-dependent detail: mirror and glass reflections come out as averaged colour, because these Gaussians have a single colour each, with no spherical harmonics.

Changes compared to question 6:

  1. Initialization that covers the scene. The default random init samples means from a ball of radius about 0.2 at the origin, so Gaussians had to travel more than 1 unit to reach the outer objects. Instead, I sample means uniformly in a ±1.2 cube. The size comes from the scene scale, not fitted to the objects; empty space gets cleaned up by step 5. Each scale starts at the mean distance to its 3 nearest neighbours, so the cloud covers space without gaps.
  2. Low initial opacity (0.1 instead of about 1). With opacity near 1, a random cloud forms a solid wall: the front Gaussians block all gradient to the ones behind. Starting translucent, as in 3DGS, lets every Gaussian receive gradient, and the optimizer then makes the useful ones opaque.
  3. Anisotropic Gaussians. Quaternions are trained too, so each Gaussian can stretch along a surface and the sphere silhouettes and flat bases are covered with fewer Gaussians.
  4. L1 + SSIM loss (0.8 L1 + 0.2 D-SSIM, as in 3DGS), with my own differentiable SSIM using an 11×11 Gaussian window. The SSIM term rewards local structure and contrast, which L1 alone doesn't on an image that is mostly black.
  5. Simplified adaptive density control (relocation). Every 100 iterations from 300 to 2,000, Gaussians whose opacity has dropped below 0.005 are moved to where they are needed. Sources are sampled in proportion to each Gaussian's accumulated position gradient. Each source is split as in 3DGS: the moved Gaussian is placed at a random offset inside the source's footprint, and both are shrunk by 1.6×. Their Adam moments are reset. The total count stays fixed, so the optimizer's parameter tensors never change shape. 2,801 Gaussians were moved in total, about 280–490 per round early on and about 100 per round later, as the empty space was used up.
  6. Learning-rate schedule and longer training. Means start 4× higher than in question 6 so they can cross the scene, then decay exponentially to 1% of that, so the end of training is fine refinement instead of jitter. I trained for 3,000 iterations instead of 1,000.