Assignment 3: Volume Rendering, NeRF and 3D Gaussian Splatting

16-825, Fall 2026 · Zijin Ye (Andrew ID: zijiny)
All results below were produced with the submitted code on an RTX 5070 Laptop GPU (8 GB).

1. Differentiable Volume Rendering

1.3 Ray sampling

Pixel grid visualization
Pixel grid (vis_grid)
Ray direction visualization
Ray directions (vis_rays)

1.4 Point sampling

Sample points along rays
Stratified sample points along the rays of the first camera

1.5 Volume rendering

Spiral rendering of the box
Spiral rendering of the box SDF volume
Depth map
Depth (normalized by its maximum)

2. Optimizing a Basic Implicit Volume

Random ray sampling picks batch_size pixels with a random permutation of the pixel grid, and the loss is the MSE between rendered and ground-truth colors.

After training on train_box:

Optimized box
Spiral rendering of the optimized box

3. Optimizing a Neural Radiance Field

Architecture. The 3D position is encoded with 6 harmonic frequencies (39 dims) and passed through a 6-layer, 128-unit ReLU MLP that re-injects the embedding at layer 3 (append_xyz: [3]). A linear layer followed by a ReLU predicts the density (non-negative), and a small 64-unit hidden layer followed by a sigmoid predicts RGB. Following the README, this model is not view dependent. It is trained with an MSE loss on 1024 random rays per step, 128 uniformly spaced samples per ray in [2, 6], for 250 epochs at 128×128.

NeRF lego spiral rendering
NeRF on lego (128×128, 250 epochs)

4.1 View Dependence

Implementation (view_dependence: True in nerf_materials*.yaml). As in the NeRF paper, density depends on position only. The color head receives a 128-dimensional bottleneck feature from the position trunk, concatenated with a harmonic embedding of the normalized ray direction (2 frequencies, 15 dims), and maps it through a single 64-unit hidden layer to RGB. The direction enters only after the density has been computed, and only through one shallow layer. So it can modulate the color of a point (specular highlights, reflections), but it cannot change the geometry or memorize a separate appearance for every training view.

Results on materials (python volume_rendering_main.py --config-name=nerf_materials: 128×128, 250 epochs, lr 1e-4): 23.73 dB mean PSNR on the 200 held-out test views (evaluate.py). The metallic and glossy spheres show highlights that move across their surfaces as the camera orbits, which a position-only color cannot produce.

View dependent NeRF on materials
View-dependent NeRF on the materials scene
View independent NeRF on materials
Same model and settings without view dependence (for comparison)

Comparison with the same model without view dependence (same seed and training settings; PSNR from evaluate.py):

ModelTrain viewsVal viewsTest viewsTrain − test gap
View-independent23.4723.3623.020.45 dB
View-dependent24.4124.2323.730.68 dB
View dependent vs independent comparison
Same orbit frames. Top: view-dependent. Bottom: view-independent. Without view dependence, the moving highlights are baked into the sphere surfaces as a mottled, speckled texture. With it, shading is smooth and the highlights on the metallic spheres are brighter and coherent.

Trade-offs between view dependence and generalization. View dependence is needed for non-Lambertian materials: without it, a highlight that moves with the camera can only be explained by baking an average highlight into the surface color, or by inventing semi-transparent geometry (floaters) that reproduces it in the training views. However, every extra degree of freedom given to the direction lets the network fit the training views with view-specific colors instead of consistent geometry. With few or unevenly distributed views, a highly view-dependent model can reproduce each training image while producing wrong colors, or even wrong shapes "hidden" behind direction-specific color, from novel viewpoints. This is the shape-radiance ambiguity discussed in NeRF++. That is why the direction branch is kept low-capacity (few frequencies, one layer, after the density). Stronger view dependence (more frequencies, a deeper branch, the direction fed into the trunk) lowers the training loss but tends to hurt held-out views.

The measurements above show both sides of the trade-off on a small scale. View dependence improves every split, so this low-capacity branch generalizes. But the gain is largest on the training views (+0.94 dB) and smaller on novel test views (+0.71 dB), and the train-test gap grows from 0.45 to 0.68 dB. Part of the extra capacity is spent fitting view-specific appearance of the training images that does not transfer to new viewpoints.

4.2 Coarse/Fine Sampling

Implementation (configs/nerf_lego_coarse_fine.yaml):

Density activation. With the ReLU density of Q3, training of this model got stuck: both networks rendered pure black from the first iterations and never escaped (their MSE equaled the error of an all-black image). This is the dead-ReLU failure mode on black-background scenes, and it also hit one unseeded materials run. For this experiment I therefore use a shifted softplus density, softplus(x − 1), as in mip-NeRF (density_activation: softplus); it cannot get stuck at zero. The single-network baseline below uses the same activation, so the comparison isolates the sampling strategy.

python volume_rendering_main.py --config-name=nerf_lego_coarse_fine
python volume_rendering_main.py --config-name=nerf_lego +implicit_function.density_activation=softplus   # baseline
Lego, 128×128, 250 epochsTest PSNR (200 views)ParametersTraining step (1024 rays)Render one 128×128 image
Single NeRF, 128 uniform samples25.73 dB101k31.4 ms144 ms
Coarse/fine, 64 stratified + 128 importance samples25.99 dB202k54.1 ms (1.7×)312 ms (2.2×)

Timings are from benchmark.py on an otherwise idle RTX 5070 Laptop GPU.

Single network NeRF
Single network (baseline)
Coarse/fine NeRF
Coarse/fine (fine network output)
Single vs coarse/fine comparison
Same frames. Top: single network. Bottom: coarse/fine. The coarse/fine renders are slightly sharper: the track treads, the edges of the bucket and the cracks on the base plate are better defined.

Trade-offs. Coarse/fine sampling concentrates the fine network's samples where the coarse pass found the surface, so most fine samples land on the object rather than in empty space. This improves quality (+0.26 dB here, with visibly crisper edges). It costs twice the parameters (two MLPs), 1.7× the training time and 2.2× the rendering time: each ray now needs 64 coarse and 192 fine network evaluations plus a sequential dependency between the passes, compared with 128 evaluations for the baseline. At this low resolution and with a tight depth range, 128 uniform samples are already about 0.03 units apart, so the baseline is not badly undersampled and the gain is modest. The benefit grows with resolution, larger depth ranges and thin structures, where uniform sampling either misses fine detail or needs many more samples. That is why NeRF uses hierarchical sampling instead of simply increasing the number of uniform samples. Stratified jitter of the coarse samples also helps by exposing the networks to continuous positions instead of a fixed grid, which reduces aliasing.

4.3 VolSDF

Implementation.

Intuition for α and β. α is the density of the solid interior: deep inside the object σ → α, and at the surface σ = α/2. It sets how quickly a ray becomes opaque once it enters the object. β is the width of the transition from empty space (σ ≈ 0) to solid (σ ≈ α) around the zero level set. Small β gives an almost binary, sharp density step at the surface; large β smears density into a thick shell on both sides of the surface.

1. How does high β bias the learned SDF? Low β? With high β, there is significant density in front of the true surface, so rays accumulate opacity and color before they reach the zero level set. The photometric loss is then satisfied by a fuzzy shell, and the zero level set is not forced onto the visible surface. It ends up offset and over-smoothed, and thin structures get washed out. With low β, density is concentrated at the zero level set, so the rendered surface and the SDF surface coincide and the bias becomes small. The cost is that density, and with it the gradient, is essentially zero away from the surface.

2. Is an SDF easier to train with low or high β? High β. The density is smooth, so many samples along each ray receive non-zero weight and gradients reach the SDF even where the current surface estimate is far from the true one. With low β, only samples within a few β of the current surface contribute. With a fixed number of uniform samples, a very thin shell can even fall between samples, so gradients are sparse and noisy and optimization stalls more easily.

3. Is an accurate surface more likely with low or high β? Low β, for the reason given in question 1. When β → 0, the density converges to α·1[d < 0], an exact indicator of the solid, and the depth at which a ray saturates converges to the SDF zero crossing. VolSDF's opacity error bound also shrinks with β. In practice a moderately small β, used together with the eikonal term, works best: the eikonal term keeps the field a proper distance function, so a small β still yields a well-defined surface.

Results

Best result: α = 33.3, β = 0.03 (α = 1/β, eikonal weight 0.02, 128 samples per ray, 250 epochs):

python -m volume_rendering_main --config-name=volsdf_surface renderer.alpha=33.3 renderer.beta=0.03
VolSDF geometry, beta 0.03
Extracted surface (marching cubes of the SDF zero level set), β = 0.03
VolSDF color rendering, beta 0.03
Volume-rendered color, β = 0.03

Default settings, for comparison: α = 10, β = 0.05

VolSDF geometry, beta 0.05
Extracted surface, β = 0.05
VolSDF color rendering, beta 0.05
Volume-rendered color, β = 0.05

Why these settings work better. With the default β = 0.05, the color renders are already good, but the extracted surface has a bloated "skirt" around and below the base plate. In that region the density is spread over a wide band, so the photometric loss is satisfied by a thick, fuzzy shell. Some of that geometry is only ever seen against the black background (or never seen at all, since all cameras are in the upper hemisphere), so nothing pulls the zero level set onto the true surface. This is the high-β bias described in question 1. Lowering β to 0.03 makes the density step at the surface sharper, so the rendered surface must coincide with the SDF zero level set. The base plate becomes a crisp rectangle, and thin parts such as the bucket arm and the tracks are recovered better. Setting α = 1/β, as in the VolSDF paper, keeps the interior opaque enough that a ray saturates within roughly one β of the surface. I did not go lower than 0.03: with 128 uniform samples over [2, 6], the sample spacing is about 0.031, and much smaller β would let the density shell fall between samples (question 2). The geometric initialization and the eikonal term keep training stable at this β.

5. 3D Gaussian Rasterization

The rasterizer in GS/model.py is fully vectorized over the N Gaussians in a mini-batch:

Rendering of pre-trained chair gaussians
Pre-trained chair Gaussians rendered with my rasterizer (RGB | depth | silhouette)

About the unit test. unit_test_gaussians.py reports 2/4. Tests 1 (3D covariance) and 3 (2D means) pass. Tests 2 and 4 fail because the reference values were computed with W = camera.R, while my code uses W = camera.Rᵀ. PyTorch3D uses row vectors (X_view = X_world · R + T), so the world-to-camera rotation in the column-vector form J W Σ Wᵀ Jᵀ is Rᵀ, not R. With W = R, every anisotropic Gaussian is rotated by the wrong matrix. The figure below shows the same three views rendered both ways: the unit-test convention (left) produces the hairy streaks that are also visible in the provided reference images, while Rᵀ (right) renders cleanly. Test 4 fails only because it reuses the 2D covariances of test 2. Changing that single line in compute_cov_2D back to camera.R makes all four tests pass.

Rotation convention comparison
Left: W = R (unit-test convention). Right: W = Rᵀ (used).

6. Training 3D Gaussian Representations

make_trainable sets requires_grad on the means, scales, colors and opacities (plus the quaternions for anisotropic Gaussians). The optimizer is Adam with one parameter group per attribute. The forward pass renders the training view with Scene.render on a black background, and the loss is the L1 difference to the ground-truth image.

ParameterLearning rateReason
Opacities (pre-sigmoid)0.05Initialized at logit 8 (opacity ≈ 1); a large step size lets them drop quickly where Gaussians should be transparent.
Scales (log)0.005Log-space, so 0.005 is a ~0.5% relative change per step; larger values make Gaussians balloon.
Colors0.02RGB in [0, 1] starts random; it has to move fast but stay stable.
Means0.0005The point-cloud initialization is already accurate; the scene spans about 5 units, so this allows sub-pixel refinements without the points drifting.
Truck training progress
Training progress (top: current renders, bottom: ground truth)
Truck final renders
Renders of the training views after training

7.1 Rendering Using Spherical Harmonics

colours_from_spherical_harmonics evaluates the 16 real SH basis functions (degrees 0 to 3) for every Gaussian at once, as an (N, 16) matrix. Each basis value multiplies the corresponding RGB coefficients of the (N, 16, 3) SH tensor, and the results are summed, then offset by 0.5 and clamped at 0, exactly as in the original implementation. The view direction of each Gaussian is the normalized vector from the camera center (camera.get_camera_center()) to its mean. render.py --use_sh switches the renderer from the DC colors to these view-dependent colors.

Rendering with DC colours only
Q5.5: DC component only
Rendering with spherical harmonics
Q7.1: all spherical harmonic components

Side-by-side comparisons (same view)

Comparison view 3
View 3. With SH the green fabric is darker and more saturated, with shading that varies across the backrest and seat. With DC colors it looks uniformly bright and flat. The DC term is the average color over all viewing directions, so it washes out direction-dependent shading. The carved wooden frame also gains contrast with SH.
Comparison view 7
View 7. The side of the backrest, seen at a grazing angle, is much darker with SH, as fabric would be when viewed edge-on. The DC render keeps the same brightness regardless of angle. The brocade pattern on the seat is also more defined with SH.
Comparison view 10
View 10. The clearest difference: the gold embroidery on the seat shows bright, metallic highlights against darker green silk with SH. The DC render blends threads and fabric into a single light green. This is a specular, view-dependent effect that a single color per Gaussian cannot represent.

7.2 Training on a Harder Scene

Both setups are in GS/train_harder_scene.py:

cd GS
python train_harder_scene.py --baseline --num_itrs 1000 --gaussians_per_splat 2500   # baseline (Q6 setup)
python train_harder_scene.py --num_itrs 5000                                         # improved approach

Baseline. Identical to Q6: 10,000 isotropic Gaussians with init_type="random" (means ~ N(0, 0.2²)), the same learning rates, L1 loss and 1000 iterations.

Modifications in the improved approach, compared to the Q6 setup:

  1. Anisotropic Gaussians: per-axis scales and trainable quaternions (lr 0.001), so flat or elongated structure can be represented without many tiny isotropic blobs.
  2. Scene-covering initialization: still random, but the default N(0, 0.2²) puts every Gaussian in a small blob at the origin, while the materials spheres lie within about 1.3 units of it. I measured this extent by carving a 3D grid against the non-black foreground of the training images; only the radius was used, not the carved points. Means are therefore drawn uniformly inside a ball of radius 1.3, scales are set from the nearest-neighbor spacing, and opacities start at 0.1 instead of ≈1, so that early Gaussians do not occlude each other.
  3. Loss: 0.8·L1 + 0.2·(1 − SSIM), as in the original paper, using a differentiable SSIM with an 11×11 Gaussian window.
  4. Learning-rate schedule: the means' learning rate decays exponentially from 1.6e-3 to 1.6e-5 over training (large moves early, fine refinement late). The other attributes keep constant rates.
  5. Simplified adaptive density control (DensityController): every 100 iterations between iterations 500 and 3000, Gaussians with a large average positional gradient are densified. Small ones are cloned; large ones are split into two samples drawn from the original Gaussian, with scales divided by 1.6. Nearly transparent (opacity < 0.005) or oversized Gaussians are pruned, capped at 40k Gaussians. Adam's moment estimates are carried over for surviving Gaussians.
  6. Longer training: 5000 iterations.
  7. Tiled splatting (Scene.splat_tiled), which makes the above affordable. Each Gaussian is evaluated only on the 16×16 tiles its footprint can reach, within radius √(2 ln(255·o) λmax), beyond which the template already zeroes α. Tiles are processed in groups with similar Gaussian counts (≤ 2²⁴ tile×Gaussian×pixel entries each), and each group is gradient-checkpointed, so memory stays bounded as densification adds Gaussians. The output is identical to the dense splat: on a test scene, images and all parameter gradients match to fp32 rounding (≤ 4e-5 against gradients up to 44), with one group or many.

Results

Learning rates. Baseline: the Q6 values (opacities 0.05, scales 0.005, colors 0.02, means 0.0005). Improved: opacities 0.05, scales 0.005, colors 0.01, quaternions 0.001, means 1.6e-3 decaying to 1.6e-5.

MethodIterationsFinal #GaussiansPSNRSSIM
Baseline (Q6 setup, random init)100010,00018.3900.675
Improved500039,91925.1170.911

Comparison. The baseline never recovers from its initialization. All Gaussians start in a small blob at the origin, and with isotropic Gaussians, a small learning rate for the means and no way to add Gaussians, they only spread a little. The progress GIF shows a blurry cluster of spheres in the middle of the frame, while the outer spheres are never reconstructed (18.4 dB). The improved approach covers the scene from the start and densifies wherever the reconstruction error is high. By the end, the renders match the ground truth closely, including the specular and patterned spheres (+6.7 dB PSNR, SSIM 0.675 → 0.911). The full 5000-iteration improved run, including the final renders and evaluation, took 28 minutes while sharing the GPU with another job. The dense baseline took 38 minutes for its 1000 iterations with 10k Gaussians.

Baseline training progress
Baseline training progress over 1000 iterations (top: renders, bottom: ground truth; one frame per 40 iterations)
Improved training progress
Improved approach training progress over 5000 iterations (one frame per 200 iterations)
Baseline final renders
Baseline final renders
Improved final renders
Improved final renders

Implementation notes