16-825 Assignment 3: Volume Rendering, NeRF and 3D Gaussian Splatting

Yi Zhao (yizhao5). All training and rendering ran on one NVIDIA H100. Every number on this page is read from the metrics files written by the runs; PSNR for the NeRF models is measured on the 100 held-out validation views of each scene.

1234.14.24.3567.17.2

A1. Differentiable volume rendering

1.3 Ray sampling

Pixel coordinates are mapped to [-1, 1] with the first and last pixel at exactly -1/+1, the convention sample_images_at_xy uses (grid_sample with align_corners=True), so a ray and the ground-truth colour it is compared with refer to the same pixel. Each NDC point is lifted to depth 1, unprojected with camera.unproject_points, and the ray direction is the normalised vector from the camera centre to that point.

xy_grid (mine)
rays (mine)
grid (reference)
rays (reference)

1.4 Point sampling

StratifiedRaysampler places n_pts_per_ray depths uniformly between near and far. While training, each depth is jittered uniformly inside its bin (NeRF's stratified sampling), so the network sees every depth over the course of training instead of a fixed grid; at evaluation the bin edges are used so renders are deterministic.

sample points, first camera (mine)
reference

1.5 Volume rendering

Weights are wi = Ti(1 - exp(-σiΔti)) with Ti = ∏j<i exp(-σjΔtj) computed by a cumulative product (T1 = 1). Colour is ∑ wi ci; depth is the expected termination depth ∑ wi ti, shown normalised by its maximum.

part_1.gif (mine)
depth, normalised (mine)
reference
reference depth

A2. Optimising a basic implicit volume

Random rays: a random permutation of all pixel coordinates, first batch_size kept (sampling without replacement). Loss: mean squared error between rendered and ground-truth colours.

Box center after training: (0.25, 0.25, 0.00)
Box side lengths: (2.01, 1.50, 1.51)
part_2.gif (mine)
reference

A3. Neural radiance field

Architecture: positional encoding of x (6 frequencies) → 6-layer MLP, 128 units, ReLU, with the encoding re-injected at layer 3 → density head (Linear + ReLU) and colour head (Linear-ReLU-Linear-Sigmoid). Loss: MSE. Settings from nerf_lego.yaml: 250 epochs, 1024 rays per step, 128 samples per ray, 128×128 images.

Validation PSNR: 25.00 dB over 100 held-out views; training took 2.9 min (25000 steps).
part_3.gif (mine)
reference

One thing that had to be fixed first. With PyTorch's default Linear initialisation my first NeRF never trained: the loss stayed flat for 140 epochs. A diagnostic showed the density was exactly zero at every sample from step 0. The default init shrinks the activations at every layer, so after the 6-layer trunk the features are nearly constant and the density takes the sign of its bias; with a negative bias the ReLU outputs 0 everywhere, the render is black, and no gradient reaches the density again. He initialisation for the ReLU layers plus a small positive density bias fixes it (same architecture, same ReLU).

A4.1 View dependence

The viewing direction is encoded with 2 harmonic frequencies and concatenated with the 128-d bottleneck features after the density head, so only the colour head sees it (Linear(128+15, 64)-ReLU-Linear-Sigmoid). Density, and therefore geometry, still depends on position alone, as in NeRF. Trained on the materials scene with nerf_materials.yaml (128×128) and nerf_materials_highres.yaml (400×400).

Materials, validation PSNR (dB)no view dependenceview dependencegain
128 × 12823.8024.57+0.77
400 × 40023.8924.77+0.88
view dependence, 400×400
no view dependence, 400×400

The materials scene is made of glossy and metallic spheres whose highlights and reflections move with the viewer, which a position-only colour can only average out. Giving the colour head the direction raises the validation PSNR at both resolutions. The visual difference is subtle at this resolution; it is clearest in the highlights on the metallic and glossy spheres in the held-out views below (each panel: prediction | ground truth, 128×128 models of the next table).

validation view 0, no view dependence
validation view 0, view dependence
validation view 10, no view dependence
validation view 10, view dependence

Trade-off between view dependence and generalisation

To make the trade-off measurable, the same 128×128 setup was trained three times and evaluated on all 100 training views and all 100 held-out views. The third model gives the direction to the trunk as well (view_dependence=early), so even the density can change with the viewpoint.

Modeltrain PSNRvalidation PSNRgap
no view dependence24.0023.800.20
direction in the colour head (NeRF)24.7624.570.19
direction in the trunk as well25.3224.910.40

Reading the table. Putting the direction into the colour head (NeRF) improves both the training views (24.76 vs 24.00 dB) and the held-out views (24.57 vs 23.80 dB) while the train-validation gap stays the same (0.19 vs 0.20 dB): the extra freedom is spent on real view-dependent appearance, not on memorising pixels. Letting the direction into the trunk too, so that density can also change with the viewpoint, fits the training views best (25.32 dB) and is even a little better on the held-out views here (24.91 dB), but its gap doubles to 0.40 dB. That growing gap is the signature of overfitting.

The trade-off. View dependence lets a model explain what one colour per point cannot (highlights, reflections), but every degree of freedom given to the direction can also explain a training pixel without getting the geometry right. Each surface point is observed from only a handful of directions, and a model whose density depends on the direction no longer has to make the views agree on one 3D shape. On this scene the 100 training cameras surround the object densely and the held-out cameras lie between them, so interpolation still works and the extra capacity is not yet harmful on average; with fewer views or cameras outside the training distribution, the model with the larger gap is the one expected to break first. NeRF's design keeps the benefit and limits the risk: density stays independent of the direction (geometry must be consistent across views), the direction only enters a small late colour head, and it is encoded with few frequencies (2 vs 6 for position), so colour can only vary smoothly with direction.

A4.2 Coarse/fine sampling

Two NeRFs of the Q3 architecture. The coarse one is evaluated at 64 stratified samples; its weights (without the two end bins) define a piecewise-constant PDF along each ray, from which 128 more depths are drawn by inverse-CDF sampling (deterministic quantiles at test time). The fine network is evaluated on the union of the 64 + 128 sorted depths. Both networks are trained with the photometric loss; the sampled depths are detached, so the coarse network learns only from its own rendering. Same lego setup as Q3 (250 epochs, 128×128).

Legoval PSNR (dB)train ms / steprender ms / 128² imageparameters
Q3: uniform, 128 samples25.007.126117,764
uniform, 192 samples25.349.334117,764
coarse 64 + fine 12825.5612.346235,528
coarse/fine (fine network)
uniform 128 (Q3)

Quality. Coarse/fine is the best of the three: +0.56 dB over Q3. Part of that is simply more samples, so the fair comparison is uniform sampling with the same 192 evaluations per ray: coarse/fine is still +0.23 dB ahead, because its 128 extra samples are placed where the coarse weights are large, i.e. around the surface, instead of in empty space.

Speed. A training step costs 1.74× Q3 and 1.31× the equal-sample uniform model, and rendering is slower by a similar factor: the fine pass cannot start until the coarse pass has produced its weights, the two networks double the parameter count, and the coarse samples are evaluated twice (once per network). The gain per sample is real but modest on this small, compact scene at 128×128; it grows when most of the ray is empty space or the resolution is high, where uniform samples are mostly wasted.

A4.3 VolSDF

Distance network. Positional encoding (6 frequencies) → 6 linear layers of 128 units with Softplus(β=100) activations → 1 signed distance + a 128-d geometry feature. The distance is left unbounded (unlike density it must be able to go negative). Softplus keeps the network smooth, which matters for the Eikonal term (a ReLU network has piecewise-constant gradients). Weights use the geometric initialisation of SAL, so training starts from the SDF of a sphere of radius 1 instead of a random field. Colour network. (geometry feature, encoded position) → Linear-ReLU-Linear-Sigmoid. Eikonal loss: mean of (‖∇d‖ - 1)² at random points in the bounding box, weight 0.02. SDF to density (VolSDF Eq. 2-3): σ(x) = α Ψβ(-d(x)), with Ψβ the CDF of a zero-mean Laplace distribution of scale β.

What α and β do. α is the density deep inside the object (the density at the surface itself is α/2): it sets how opaque the solid is, i.e. how quickly a ray is absorbed once it enters. β sets how wide the transition is: density goes from ~0 to ~α over a band of a few β around the zero level set. Small β is a sharp, almost binary occupancy; large β a soft shell.

1. How does high β bias the learned SDF? Low β? With a high β the density is spread over a thick band around the zero level set, so what the renderer sees is a soft shell rather than the surface. The photometric loss can then be satisfied by a smooth, rounded SDF whose zero set does not have to sit where the true surface is; thin parts and sharp edges are smoothed away or merged (compare the β = 0.3 geometry below, which is visibly rounder and loses the tread and shovel detail). With a low β the density is close to a step at d = 0, so the rendered surface and the zero level set coincide, but only points within a few β of the current surface receive any gradient.

2. Easier to train with low or high β? High β. The density, and hence the loss, varies smoothly over a wide band, so samples far from the current surface still get a gradient and a coarse uniform sampler always has several samples inside the transition. With low β the transition is narrower than the sample spacing (Δt = 0.031 here) and most samples see either ~0 or ~α, which gives sparse, noisy gradients.

3. More accurate surface with high or low β? Low β, provided it can still be trained: VolSDF bounds the opacity error by a term proportional to β, and a sharp transition pins the rendered surface to the zero level set. In my sweep, however, β is not the only knob that matters, which is the more useful observation. At α = 10, lowering β from 0.3 to 0.02 helps (20.95 → 22.41 dB) and then stops helping (22.31 dB at β = 0.01). The limit is α: one sample of fully occupied space absorbs only 1 - exp(-αΔt) = 27% of the light at α = 10, so a ray travels several samples into the object before it is stopped and the rendered surface lies inside the zero set no matter how sharp β is (this is the 'skirt' of half-transparent geometry around the base). At α = 50 a sample absorbs 79%, at α = 100 96%, and every low-β run gains about 1.7-1.9 dB (22.31 → 24.05 → 24.24 dB at β = 0.01). So an accurate surface needs a sharp transition (low β) and a density high enough that the transition is opaque at the sampling resolution (high α).

αβval PSNR (dB)
100.0122.31
100.0222.41
100.0522.37
100.122.04
100.320.95
500.0124.05
500.0224.20
500.0523.97
1000.0124.24

Best setting: α = 100, β = 0.01 (24.24 dB). All runs use the volsdf_surface.yaml schedule (250 epochs, 128 samples per ray, Eikonal weight 0.02).

geometry, α=100, β=0.01
colour, α=100, β=0.01
reference geometry
reference colour

The same views for other settings (geometry):

α=10, β=0.05
α=10, β=0.3
α=50, β=0.01
α=100, β=0.01

Settings I chose and why. α = 100, β = 0.01 gives the best validation PSNR (24.24 dB) and the cleanest geometry: high α makes the surface opaque within one or two samples, and a β comparable to the sample spacing keeps the transition sharp while still being trainable with 128 uniform samples. The differences among β ≤ 0.05 at a fixed high α are small (within ~0.2 dB), while going from α = 10 to α ≥ 50 is the single largest improvement. Everything else follows volsdf_surface.yaml, with the Softplus distance network and sphere initialisation described above.

B5. 3D Gaussian rasterisation

Covariance Σ = R S ST RT (for isotropic gaussians R = I, so Σ = s²I without building a rotation). Projection: Σ' = J W Σ WT JT with the provided Jacobian; 2D means from transform_points_screen. Gaussians are sorted by view-space depth and those behind the camera dropped; α = o·exp(P), Ti = ∏j<i(1-αj) by a cumulative product, and colour, depth and silhouette are ∑ ciαiTi, ∑ ziαiTi and ∑ αiTi. The provided unit tests pass (4/4). Each frame below shows colour, depth and silhouette.

part_5_render.gif

A finding: the world-to-camera rotation in Eq. (5)

The unit test for compute_cov_2D only passes with W = camera.R. PyTorch3D, however, stores R for row vectors (xcam = xworld R + T), so the column-vector rotation that belongs in Eq. (5) is RT. The two only agree for isotropic gaussians, which is why the difference is easy to miss. For the anisotropic chair the effect is visible: with W = R the ellipses are rotated by the wrong matrix and thin parts turn into fuzzy streaks; with W = RT the legs and the carved frame are crisp. The code keeps the unit-test convention as the default (so the tests pass and the GIF above is the expected output); --transpose_view_rotation switches to RT.

frame 8: left W = R, right W = RT
frame 20: left W = R, right W = RT
full GIF with W = RT

B6. Training 3D gaussians (truck)

All gaussian attributes are made leaf tensors with gradients (quaternions are not optimised for isotropic gaussians). One Adam optimiser with a separate learning rate per parameter group; loss: L1 between the render and the image. Means live in world units and get the smallest learning rate; log-scales are also kept small because a step in log space changes a gaussian's footprint multiplicatively; colours and opacities can move fast.

lr meanslr scaleslr colourslr opacitiesiterationsPSNRSSIM
0.00050.0050.020.05100029.580.938
0.00020.0050.020.05100029.220.935
0.0010.0050.050.05100029.040.931
0.00050.0050.020.05300030.970.951
Reported run: learning rates means 0.0005, scales 0.005, colours 0.02, opacities 0.05; 3000 iterations; held-out views: PSNR 30.97, SSIM 0.951.

Lowering the mean learning rate to 0.0002 slows the geometry down (29.22 vs 29.58 dB at 1000 steps); raising the mean and colour rates makes training noisier and ends lower (29.04 dB). With the middle setting, training longer helps the most: 3000 steps reach 30.97 dB / SSIM 0.951, so 1000 steps are not yet converged on this scene. The four runs were compared on the same held-out views that are reported, so the choice among them is itself made on those views; the 3000-step run is the one shown.

part_6_training_progress.gif (top: renders, bottom: ground truth)
part_6_training_final_renders.gif

B7.1 Rendering with spherical harmonics

colours_from_spherical_harmonics evaluates the real spherical harmonics up to degree 3 (16 coefficients per channel, the same basis and constants as the official 3DGS code) at the unit direction from the camera centre to each gaussian, then adds 0.5 and clamps at 0, the same offset used for the DC-only colours.

part_7_1_render.gif (with SH)
part_5_render.gif (DC only, Q5.5; same file as in B5)

Side-by-side comparisons (same view)

frame 4: left DC only, right full SH

Seen from above, the DC-only render is a flat, uniformly light green. With the full spherical harmonics the backrest and seat are a deeper green, the gold brocade pattern stands out, and the lower part of the backrest is darker than the upper part: the shading now depends on the viewing angle instead of being baked into one average colour.

frame 12: left DC only, right full SH

Looking down onto the seat, the silk shows bright gold sheen on the woven pattern with SH, which is a view-dependent specular effect; the DC colour averages the sheen over all directions and looks washed out and lower in contrast.

frame 24: left DC only, right full SH

From underneath, the two renders are almost identical. The underside is mostly matte wood and fabric seen at a grazing angle, so the higher-order SH terms contribute little; view dependence matters where the material is glossy.

B7.2 Training on a harder scene (materials, random initialisation)

Setupiterationsfinal gaussiansPSNRSSIM
Baseline: Q6 recipe (isotropic, Q6 learning rates, L1), 1000 steps100010,00018.520.675
Baseline, trained 5000 steps500010,00018.860.708
Improved (all modifications)50008,56924.780.900
Improved without densification500010,00023.700.870
Improved with isotropic gaussians50008,63423.920.880
Improved with W = R (unit-test convention)50008,62424.390.889
Baseline (Q6 recipe): PSNR 18.52, SSIM 0.675. Improved: PSNR 24.78, SSIM 0.900 on the held-out views.
baseline: training progress
baseline: final renders
improved: training progress
improved: final renders

Why the baseline fails. The baseline uses the Q6 setup (isotropic gaussians, the Q6 learning rates, L1 loss) with init_type="random". The reported Q6 truck model was trained for 3000 steps; here the baseline is shown at 1000 steps (18.52 dB) and at 5000 steps (18.86 dB), which bracket that, so it is not just under-trained. Its random init puts all 10,000 isotropic, fully opaque gaussians in a small blob at the centre while the spheres are spread over a much larger area; the gaussians have to travel far, block each other, and many never reach an object, so the renders stay a few blurry blobs.

Modifications in the improved run (relative to Q6; the ablations remove one change at a time from the full recipe, same number of steps):

  1. Anisotropic gaussians. each gaussian gets a rotation (quaternion, learning rate 0.001) and three scales, so flat or elongated surface pieces can be covered by few gaussians. Ablation: +0.85 dB vs isotropic.
  2. Correct view rotation (W = RT). with anisotropic gaussians the projected ellipse must be rotated consistently across views (see B5); with W = R the same gaussian projects to inconsistent ellipses in different views. Ablation: +0.39 dB vs W = R.
  3. Simplified adaptive density control. every 100 steps from step 300 to 2500, the 5% of gaussians with the largest average positional gradient are cloned (if small) or split into two samples of 1/1.6 the scale (if larger than 1% of the scene extent); gaussians with opacity below 0.005 or larger than 20% of the extent are pruned (capped at 30,000). Because our rasteriser does not expose the 2D mean, the gradient is that of the 3D mean, whose scale is scene dependent, so a quantile replaces 3DGS's fixed threshold. Ablation: +1.08 dB vs no densification.
  4. Better random initialisation. means uniform in a ball of radius 1.5 covering the objects (instead of a N(0, 0.2²) blob at the centre), scales from the nearest-neighbour spacing, and opacity 0.1 (instead of ~1) so early gaussians do not occlude each other.
  5. L1 + D-SSIM loss. (1 - 0.2) L1 + 0.2 (1 - SSIM), the loss of 3DGS, which rewards local structure and contrast and not only per-pixel colour.
  6. Learning-rate schedule and longer training. 5000 steps; the mean learning rate decays exponentially from 0.0016 to 1.6e-05 (large moves early while gaussians travel from the random init to the objects, fine positioning later).

Each configuration was trained once (one seed), so differences of a few tenths of a dB between ablations should be read as indicative; the gap between the baseline and the improved recipe (6.3 dB) is far larger than that.

Use of generative AI

Anthropic Claude was used to explain course concepts and to polish the writing.