Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

16-825: Learning for 3D Vision · Kashu Yamazaki

A1. Differentiable Volume Rendering

1.3 — Ray sampling

Pixel centres are placed on a W×H grid spanning [−1, 1] in NDC. Each pixel is lifted to depth 1 with camera.unproject_points; the ray origin is the camera centre and the direction is the normalized vector from the origin to that point.

Pixel grid visualization.
xy grid
Ray direction visualization.
Ray directions (|d| as RGB)

1.4 — Point sampling

64 depths evenly spaced in [near, far] are shared by every ray: p = o + t·d.

Sample points along the rays of the first camera.
Sample points from the first camera (every 16th ray).

1.5 — Volume rendering

Weights are wi = Ti(1 − e−σiΔti) with Ti = ∏j<i e−σjΔtj (an exclusive cumulative product, so T1 = 1). Colour and depth are the weighted sums of per-sample colour and sample distance.

Rendered box, rotating.
Spiral render
Rendered depth of the box.
Depth, normalized by its maximum

A2. Optimizing a Basic Implicit Volume

Training rays are a random subset (randperm) of the pixel grid, and the loss is the MSE between rendered and ground-truth colours. After 1000 epochs:

xyz
Box centre0.250.250.00
Side lengths2.011.501.50
Optimized box, rotating.
Optimized volume; matches the TA reference.

A3. Neural Radiance Field

The positional encoding uses 6 frequencies on xyz. A 6-layer, 128-unit ReLU MLP with the encoding re-concatenated at layer 3 produces a feature; a linear head with ReLU gives density, and a linear 128→128 layer followed by a 128→64→3 head with a sigmoid gives colour. Training follows nerf_lego.yaml: 250 epochs of 1024 rays per image, 128 samples per ray, Adam at lr 5e-4 decayed by 0.8 every 50 epochs, MSE loss. Test PSNR (200 held-out views, 128×128) is 24.09 (train 24.52).

Note on NeRF metrics. The provided training loop saves a checkpoint only every 50 epochs, so the last saved model is from epoch 200. All PSNR values in A3 and A4 are computed from these epoch-200 checkpoints with eval_nerf.py; the GIFs are rendered during training at epoch 240.

NeRF render of the lego bulldozer, rotating.
Lego bulldozer, 128×128.

A4. NeRF Extension: View Dependence (4.1)

The colour head additionally takes the harmonic embedding (2 frequencies) of the ray direction, concatenated with the position feature. Density still depends on position only, as in NeRF, so the geometry is shared across views and only the emitted colour can change with direction. Same schedule as the lego run, at lr 1e-4 per nerf_materials.yaml.

MaterialsTrain PSNRVal PSNRTest PSNRTrain − test
No view dependence, 128²23.4723.3523.030.44
View dependence, 128²24.2724.1223.640.63
View dependence, 400²24.4024.2623.790.61
Materials scene without view dependence.
No view dependence
Materials scene with view dependence.
View dependence
Materials scene with view dependence at 400 by 400.
View dependence, 400×400. Specular highlights are visible on the metallic spheres.

View dependence lets the network represent the specular highlights of the metallic and glossy spheres, which a position-only colour can only average into a dull, view-independent tint; it raises test PSNR by 0.6 dB. The cost is a larger train–test gap (0.44 → 0.63 dB): with direction as an input the network can also explain a training pixel by a colour that is only correct from that particular view, instead of by better geometry. The design follows NeRF to limit this: density does not depend on direction, the direction uses only 2 frequencies, and it enters late, through a small head. I did not test variants with more direction capacity (direction fed to the density, or more frequencies); the expected risk is that the network memorizes the training views and produces wrong colours or floaters from new viewpoints. All numbers are from a single run each, so the 0.6 dB gain and the 0.2 dB change in the gap carry run-to-run noise.

B5. 3D Gaussian Rasterization

Covariances are Σ = R S ST RT (or s2I for isotropic Gaussians) and project as Σ' = J W Σ WT JT; means are projected with transform_points_screen. Gaussians behind the camera are dropped, the rest sorted by view-space depth, and colour, depth and mask are the αT-weighted sums of colour, depth and 1.

Note on W. PyTorch3D stores R for row vectors (xview = xworldR + T), so the world-to-view rotation in the column-vector formula is W = RT. The provided unit tests 2 and 4 expect W = R; with RT they fail while tests 1 and 3 pass. Projecting 200k samples of a 3D Gaussian through the camera gives an empirical 2D covariance that matches W = RT to three decimals; with W = R one diagonal entry is 3.4× too large and the off-diagonal term has the wrong sign, and with W = R the chair's anisotropic splats render as visibly mis-oriented streaks. The renders below use RT.

Pre-trained chair Gaussians: colour, depth and mask, rotating.
Pre-trained chair: colour, depth and silhouette.

B6. Training 3D Gaussians

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.02
Means0.0005

Adam, L1 loss, 1000 iterations, isotropic Gaussians initialized from the provided 10k-point cloud. On the 5 held-out test views of the truck dataset: PSNR 29.49, SSIM 0.939.

Training progress on the truck scene.
Training progress: renders (top) and ground truth (bottom).
Final renders of the training views.
Final renders of the training views.

The full-batch rasterizer evaluates every Gaussian at every pixel and keeps all intermediates for the backward pass; the README estimates about 15.5 GB for this, more than the 10 GB of the RTX 3080 used here. Each mini-batch splat is therefore wrapped in torch.utils.checkpoint (1000 Gaussians per splat), which recomputes its intermediates during backward instead of storing them. The result is unchanged; training is slower.

B7. Extensions

7.1 — Rendering with spherical harmonics

Colours are evaluated from all 16 SH coefficients per channel (degree 3), using the normalized direction from the camera centre to each Gaussian, then offset by 0.5 and clamped at 0.

Chair rendered with the DC term only.
DC only (Q5.5)
Chair rendered with full spherical harmonics.
Full spherical harmonics

Each comparison below shows the DC-only render on the left and the full-SH render on the right, from the same camera.

DC-only and full-SH renders of the chair, frame 4.
Frame 4. With SH the backrest fabric is darker and shaded, with brighter highlights on the embroidered pattern and the gilded frame; the DC render is a flat, uniformly lighter green.
DC-only and full-SH renders of the chair, frame 12.
Frame 12. The seat cushion turns a deep glossy green with the gold pattern standing out under SH. The DC term averages the cushion's appearance over all directions, so it loses this contrast.
DC-only and full-SH renders of the chair, frame 20.
Frame 20 (from below). The matte underside and wooden legs look nearly the same either way, which suggests these diffuse surfaces have little view-dependent appearance for the higher-order coefficients to add.

7.2 — Training on a harder scene

Materials scene at 128×128, 10k Gaussians with init_type="random", evaluated on the 100 validation views. The baseline is the Q6 setup unchanged (isotropic, L1, the Q6 learning rates, 1000 iterations; train_harder_scene.py --baseline).

MaterialsIterationsPSNRSSIM
Baseline (Q6 setup)100018.230.685
Improved300023.750.876
Baseline training progress on the materials scene.
Baseline training progress (every 20 iterations): renders (top), ground truth (bottom).
Improved training progress on the materials scene.
Improved training progress (every 60 iterations).
Baseline final renders of the training views.
Baseline, final renders
Improved final renders of the training views.
Improved, final renders

The default random init draws means from N(0, 0.2²), nearly opaque (sigmoid(8)) and tiny. The spheres span roughly ±1.5 units, so almost every Gaussian starts inside the centre of the scene. The baseline ends with blurry blobs clustered around the middle spheres, which suggests that at the Q6 learning rate the means do not move far enough from this start. Modifications, in the improved run:

These were applied together and not ablated one at a time (each run takes 30–85 minutes on this GPU), so the gain is not attributed to individual changes. Adaptive density control (cloning, splitting and pruning) was not used.