16-825 Learning for 3D Vision — Assignment 3

Volume Rendering, NeRF, and 3D Gaussian Splatting · Andrew ID: farnazaa

1. Differentiable Volume Rendering

1.3 Ray sampling

Pixel coordinates are generated on a W×H grid and mapped to NDC in [-1, 1]. Each pixel is lifted to the plane z = 1 in NDC, unprojected to world space with camera.unproject_points, and the ray direction is the normalized vector from the camera center to that point. Both visualizations match the TA references.

xy grid
xy_grid
ray directions
rays (|direction|)

1.4 Point sampling

64 depths are spaced uniformly between near = 0 and far = 5, and each sample point is origin + t · direction. The sample points from the first camera form the expected viewing frustum.

sample points
Sample points along rays (camera 0)

1.5 Volume rendering

For each sample, αi = 1 − exp(−σiΔti) and Ti = ∏j<i(1 − αj) (computed with a cumulative product, with T1 = 1). The weights wi = Tiαi are used for both color (∑ wici) and depth (∑ witi). The rendered GIF matches the TA GIF to within GIF compression noise (mean absolute difference ≈ 1/255 per pixel).

part 1 gif
part_1.gif
depth
Depth (normalized by its maximum)

2. Optimizing a Basic Implicit Volume

2.1 Random ray sampling

Each training step renders only a random subset of rays. torch.randperm shuffles all pixel coordinates and the first n_pixels are kept (4096 per step, no duplicates). The same coordinates are used to sample the ground-truth colors.

2.2 Loss and training

The loss is the mean squared error between rendered and ground-truth colors. Training the box's center and side lengths for 1000 epochs (Adam, lr 5e-4) gives, rounded to 1/100:

xyz
Box center0.250.250.00
Box side lengths2.011.501.50

2.3 Visualization

Spiral rendering of the optimized box. It matches the TA's part_2.gif.

part 2 gif
part_2.gif (optimized box)

3. Optimizing a Neural Radiance Field

Architecture. Positions are encoded with 6 harmonic frequencies (3 → 39 dims) and passed through a 6-layer, 128-unit ReLU MLP with a skip connection that re-injects the encoding at layer 3. A linear head followed by ReLU gives density, and a 128→64→3 head followed by Sigmoid gives color. Trained for 250 epochs on 128×128 images, 1024 rays per step, 128 samples per ray, Adam lr 5e-4.

Note on initialization. With a ReLU density, the randomly initialized network output zero density at every sample, so no gradient reached the density head and the renders stayed black. Initializing the density layer's bias to 0.1 makes density start slightly positive, and training then converges normally.

lego nerf
part_3.gif (test PSNR 23.08 dB on 20 held-out views)

4. NeRF Extensions

I implemented all three extensions (4.1 is the required one; 4.2 and 4.3 are extra credit).

4.1 View dependence

The viewing direction is encoded with 2 harmonic frequencies (3 → 15 dims) and concatenated with the 128-d trunk feature as input to the color head only. Density still depends only on position. Trained on the materials scene (nerf_materials.yaml: 250 epochs, 128×128, lr 1e-4).

materials
Materials scene with view-dependent NeRF: specular highlights move across the spheres as the camera orbits

Trade-off between view dependence and generalization. View dependence lets the model represent specular highlights and reflections, which a position-only NeRF has to blur or bake into a fixed color. The risk is overfitting: with only ~100 training views, a powerful direction input lets the network explain each training image with view-specific colors instead of learning correct geometry, which produces artifacts or floaters from novel viewpoints. The design limits this in three ways: (1) direction feeds only the color head, so geometry is shared by all views; (2) direction uses far fewer frequencies than position (2 vs. 6), so view effects stay smooth; and (3) direction enters late, through a small head, so most of the capacity goes to view-independent structure.

4.2 Coarse / fine sampling

Two NeRFs of the Q3 architecture are used. The coarse network is evaluated at 64 stratified samples. Its weights (excluding the endpoints) define a piecewise-constant PDF over the bins between sample midpoints, and 64 additional depths are drawn from it by inverse-transform sampling (random during training, evenly spaced at render time). The fine network is evaluated at the sorted union of all 128 depths. The loss is MSE(fine) + MSE(coarse), and the coarse weights are detached before sampling. For a fair comparison, the Q3 baseline uses 128 uniform samples, so the fine network sees the same number of points per ray.

ModelTest PSNR (20 views)Render time / 128×128 imageTraining speed
Q3 NeRF, 128 uniform samples23.08 dB135 ms36.9 it/s (~14 min)
Coarse/fine, 64 + 64 samples23.30 dB202 ms (1.5×)19.7 it/s (~25 min, 1.9×)
baseline vs coarse/fine
Same frame: Q3 baseline (left) vs. coarse/fine (right), upscaled 3×
coarse fine
Coarse/fine NeRF

Trade-offs. Importance sampling places the fine network's samples where the coarse pass found surfaces instead of spending most of them on empty space, which gives a small but consistent quality gain (+0.22 dB, slightly crisper edges). The cost is running two networks: 64 coarse + 128 fine evaluations per ray instead of 128, so rendering is about 1.5× slower and training about 1.9× slower. At 128×128 resolution with a compact object, uniform sampling is already reasonably dense, so the gain is modest. It matters more at higher resolution or with a tighter sample budget.

4.3 VolSDF

Implementation. NeuralSurface encodes positions with 6 harmonic frequencies and passes them through a 6-layer, 128-unit MLP. Its first output is the signed distance (no activation, since distance can be negative), and the remaining 128 outputs are a feature vector that a 2-layer color MLP maps to RGB with a sigmoid. The eikonal loss is mean((‖∇d‖ − 1)²) on random points in the scene bounds (weight 0.02). SDF to density follows VolSDF: σ = α·Ψβ(−d), where Ψβ is the CDF of a zero-mean Laplace distribution with scale β. I wrote it with exp(−|d|/β) in both branches so it cannot overflow and produce NaN gradients.

What α and β do. α is the density deep inside the object: the maximum opacity. β controls how sharply density falls off around the zero level set. Density goes from α to 0 over a band roughly a few β wide, so β acts as the "blurriness" or thickness of the surface.

  1. How does high/low β bias the learned SDF? High β spreads density over a thick band around the surface, so the rendering is insensitive to exactly where the zero level set is. The learned surface is smoothed, blobby, and loses thin structures. Low β makes density a near step function at d = 0, so the rendered surface sits exactly at the zero level set and the SDF is pushed to put it in the right place.
  2. Easier to train with low or high β? High β. Density varies gradually over a wide band, so many samples along each ray receive useful gradients even when the surface starts in the wrong place. With low β, only samples extremely close to the current surface get gradient, so optimization is unstable and can stall.
  3. More accurate surface with low or high β? Low β (once training succeeds), because the rendered geometry is then a thin shell at the true zero level set instead of a thick, blurred band.

Experiments and chosen settings. With the default settings (α = 10, β = 0.05) the color renders looked fine, but the extracted mesh had a large lumpy "floor" of spurious surface around the object. At initialization the network outputs SDF ≈ 0 everywhere, so the whole scene starts as surface, and because the background is black the network learns to paint that surface black instead of removing it. I added a sphere prior: the MLP predicts a residual on top of the SDF of a radius-1 sphere (d = MLP(x) + ‖x‖ − 1), so space far from the object starts out empty. This removed the spurious geometry. I then swept β with the prior:

SettingTest PSNRGeometry
No prior, β = 0.0521.17 dBlarge spurious surface around the object
Sphere prior, β = 0.120.47 dBclean but blobby, rounded base
Sphere prior, β = 0.0520.90 dBclean, slightly rounded base
Sphere prior, β = 0.0221.36 dBsharpest: flat rectangular base, crisp scoop

The best setting is α = 10, β = 0.02 with the sphere prior (250 epochs, lr 5e-4, eikonal weight 0.02). The trend matches the analysis above: lower β gives a sharper, more accurate surface, and the sphere prior is what makes the low-β setting trainable by giving it a sensible starting surface. VolSDF's PSNR is a little below the density NeRF's (21.4 vs. 23.1 dB) because it is constrained to a single clean surface, in exchange for well-defined geometry.

best geometry
Best: geometry (β = 0.02, sphere prior)
best color
Best: color (β = 0.02, sphere prior)
no prior geometry
No prior, β = 0.05
beta 0.05 geometry
Sphere prior, β = 0.05
beta 0.1 geometry
Sphere prior, β = 0.1

5. 3D Gaussian Rasterization

Each 3D Gaussian's covariance is Σ = R S ST RT (s²I in the isotropic case). It is projected to a 2D covariance Σ' = J W Σ WT JT, and its mean is projected to pixel coordinates with transform_points_screen. Gaussians are sorted by camera-space depth, and those behind the camera are discarded. For every pixel, α = o·exp(−½(x−μ')TΣ'−1(x−μ')), T is the cumulative product of (1 − α) over closer Gaussians, and color, depth and mask are ∑cαT, ∑zαT and ∑αT. All 4 unit tests pass, and the frame from the same camera as the reference image is pixel-identical to it.

q5 render
Pre-trained chair: color | depth | mask (DC color only)

6. Training 3D Gaussian Representations

ParameterLearning rate (Adam, eps 1e-15)
opacities (pre-sigmoid)0.05
scales (log)0.005
colours0.02
means0.0005
IterationsLossMean PSNR (test)Mean SSIM (test)
1000L129.9060.942

Means get the smallest learning rate because small moves shift a Gaussian across many pixels, while colors (initialized randomly) and opacities can change faster. To train on an 8 GB laptop GPU, I added gradient checkpointing to the mini-batched splatting in Scene.render: each mini-batch's intermediates are recomputed during the backward pass instead of stored. This lowered peak memory from ~15 GB to ~3.5 GB (run with --gaussians_per_splat 2048).

training progress
Training progress: renders (top) vs. ground truth (bottom)
final renders
Final renders of the training views

7. Extensions

7.1 Rendering Using Spherical Harmonics

Each pre-trained Gaussian stores 16 SH coefficients per color channel (degrees 0–3, 48 numbers). For every Gaussian I compute the unit direction from the camera center to its mean, evaluate the 16 real SH basis polynomials at that direction, and take the coefficient-weighted sum plus 0.5, clamped at 0 (a vectorized version of the reference get_color). As a check, keeping only the degree-0 coefficient reproduces the Q5 DC colors exactly.

DC only
Q5: DC color only
full SH
Q7.1: full spherical harmonics (degree 3)

Side-by-side comparisons of the same frame (left: DC only, right: full SH):

frame 0
Frame 0: front view
frame 14
Frame 14: oblique view from above
frame 8
Frame 8: looking down onto the seat
frame 20
Frame 20: view from underneath

7.2 Training on a Harder Scene

The materials scene is trained from random initialization (init_type="random": 10,000 Gaussians with means drawn from a normal distribution of std 0.2 around the origin) at 128×128. Both runs use gradient-checkpointed splatting on an 8 GB laptop GPU.

RunGaussiansLearning ratesLossIterationsMean PSNRMean SSIM
Baseline (Q6 setup)isotropic opacities 0.05, scales 0.005, colours 0.02, means 0.0005L1100018.1150.674
Improvedanisotropic opacities 0.05, scales 0.005, colours 0.02, quats 0.001, means 0.002 → 0.00002 (exponential decay) 0.8·L1 + 0.2·(1 − SSIM)300023.0820.854
baseline vs improved vs ground truth
End of training, same 4 views. Top: baseline. Middle: improved. Bottom: ground truth.
baseline progress
Baseline training progress (renders top, ground truth bottom)
improved progress
Improved training progress (renders top, ground truth bottom; every 3rd frame shown)
baseline final
Baseline: final renders of the training views
improved final
Improved: final renders of the training views

Comparison. The baseline never recovers the scene layout. Its Gaussians stay clumped near the origin, where they were initialized, and the objects further out appear only as faint, blurry blobs because no Gaussians reach them in 1000 iterations with a small means learning rate. The improved model places all twelve objects correctly with the right colors and visible material differences (metallic, glossy, matte). PSNR improves by about 5 dB and SSIM by 0.18. The remaining errors are mostly slightly fuzzy, spiky silhouettes and missing fine specular detail, since colors are view-independent here.

Modifications compared with the Q6 setup (all used together in the improved run; I did not have time for a per-change ablation):

  1. Wider random initialization (means std 0.2 → 0.6). The default puts every Gaussian in a small ball, while the objects are spread over roughly ±1 units. Starting the Gaussians spread over the scene means they only need to make small moves to reach an object, instead of traveling across the whole scene.
  2. Higher means learning rate with exponential decay (0.002 decaying to 1% by the end, as in the original paper's position schedule). Early on, Gaussians need large steps to move to the right objects. Later, small steps let them settle precisely without jitter. The fixed 0.0005 from Q6 was tuned for a point-cloud initialization that is already almost correct.
  3. L1 + SSIM loss (0.8 / 0.2, the weighting used by the original paper). The SSIM term rewards correct local structure (edges, contrast) and not only per-pixel color, which gives sharper object boundaries. I implemented a differentiable SSIM with an 11×11 Gaussian window and checked it against scikit-image.
  4. Anisotropic Gaussians (3 scales + a rotation per Gaussian; quaternion learning rate 0.001). Ellipsoids can flatten onto the curved object surfaces and stretch along edges, so the same 10,000 Gaussians cover the surfaces better than spheres.
  5. Longer training (1000 → 3000 iterations). Starting from random initialization needs more steps to converge. The improved run took 59 min vs. 25 min for the baseline.