16-825 Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

Andrew ID: tzuyiy

A.1 Differentiable Volume Rendering

1.3 Ray sampling

NDC pixel coordinates in [-1, 1] are unprojected to world space at depth 1; each ray direction is the normalized vector from the camera center to that point.

xy grid
xy grid
rays
Ray directions

1.4 Point sampling

Evenly spaced depths between min_depth and max_depth: origin + z · direction.

sample points
Sample points from the first camera (every 50th ray)

1.5 Volume rendering

alpha_i = 1 - exp(-sigma_i * delta_i)
T_i     = prod_{j < i} (1 - alpha_j)
w_i     = T_i * alpha_i
color   = sum_i w_i * c_i          depth = sum_i w_i * z_i
box render
Rendered box
depth
Depth, normalized by its maximum

A.2 Optimizing a Basic Implicit Volume

Trained with an MSE loss on randomly sampled pixels.

Box center(0.25, 0.25, 0.00)
Box side lengths(2.01, 1.50, 1.50)
before
Before training
after
After training
part 2
Optimized box (matches part_2.gif)

A.3 Optimizing a Neural Radiance Field

6 harmonic frequencies, then a 6-layer, 128-wide MLP with a skip connection at layer 3; ReLU density head and sigmoid color head. Default nerf_lego.yaml settings: 250 epochs, 1,024 rays, 128 samples per ray, lr 5e-4.

lego
NeRF on lego
pred
Validation view 0, prediction
gt
Ground truth
pred 33
Validation view 33, prediction

Validation PSNR 24.55 dB, SSIM 0.893.

A.4.1 View Dependence

The direction (2 harmonic frequencies) is concatenated to the trunk features and fed only to the color head (Linear → 64 → ReLU → 3 → Sigmoid), so density depends on position only. Both models use nerf_materials.yaml and differ only in this switch.

Materials, 128×128Val PSNRVal SSIM
No view dependence23.260.847
View dependence23.820.855
no vd
No view dependence
vd
View dependence
Position only, view 33
View dependent, view 33
Ground truth

Trade-offs. View dependence raised PSNR from 23.26 to 23.82 dB. The position-only model bakes whitish blotches into the copper and gold spheres; the view-dependent model is smoother, though neither captures the sharp reflections at 128×128. Too much direction capacity hurts generalization: the network can fit each training view by changing color instead of geometry. I keep it small: 2 frequencies, color head only.

A.4.2 Coarse/Fine Sampling

Two Part 3 MLPs. The coarse weights at 64 evenly spaced samples define a PDF along the ray, from which 128 more samples are drawn by inverse-CDF sampling (weights detached). The fine network renders all 192 sorted samples; loss = coarse MSE + fine MSE. Both models: 25,000 updates.

Lego, 128×128Network evals / rayTrain speed (CPU)Val PSNRVal SSIM
Single network, 128 samples (Part 3)1287.6 it/s24.550.893
Coarse/fine, 64 + 128 samples64 + 1923.9 it/s24.940.898
Single network
Coarse/fine

Trade-offs. +0.39 dB, because the extra samples go near the surface instead of empty space. The cost is two networks, 256 instead of 128 evaluations per ray, and half the training speed (3.9 vs 7.6 it/s on CPU) for the same number of updates.

A.4.3 VolSDF

A 6-layer, 128-wide MLP predicts a residual on a unit-sphere SDF, d(x) = MLP(x) + ||x|| - 1, plus features; a 2-layer color head ends in a sigmoid. Density is alpha · Psi_beta(-d) (Laplace CDF). Eikonal loss weight 0.02 on random points in [-4, 4]³.

What alpha and beta do. alpha is the maximum density, reached deep inside the object; density is alpha/2 on the surface and 0 far outside. beta is the width of this transition; as beta → 0 density becomes a step at the surface.

  1. How does high / low beta bias the SDF? High beta makes space outside the surface partly opaque, so the images are fit by a blurred volume and the SDF comes out too smooth: thin parts and edges get rounded (my beta 0.1 result). Low beta has little bias: the visible surface coincides with the zero level set.
  2. Easier to train with high or low beta? High beta. The density gradient covers a wide band, so samples far from the current surface still guide the SDF. With low beta only samples in a thin band get gradients, and rays can step over it.
  3. More accurate surface with high or low beta? Low beta, as long as it still trains: only then must the zero level set match the true surface. Here beta 0.02 gave the sharpest geometry and the best PSNR.

Results and hyper-parameters

alpha = 10; other settings from volsdf_surface.yaml.

Lego, alpha = 10Val PSNRVal SSIM
beta = 0.121.200.787
beta = 0.05 (config default)21.430.798
beta = 0.0221.590.819

Best result: beta = 0.02.

VolSDF color, beta 0.02
Geometry, beta 0.02
beta 0.1
beta 0.05
beta 0.02

Lower beta gives better metrics and sharper geometry. At 0.1 the base plate is a rounded blob and the arm merges with the body; at 0.02 the edges are sharp and the parts are separated. With the sphere init, small beta trained stably.

Effect of the sphere initialization

No sphere init, beta 0.05 (val PSNR 21.96)
Sphere init, beta 0.05 (val PSNR 21.43)

Without it, beta 0.05 converged but grew a bumpy false floor in the black background, which the color loss does not penalize, and beta 0.1 and 0.02 diverged (~13 dB). The sphere init fixes both at a cost of about 0.5 dB.

B.5 3D Gaussian Rasterization

Covariances R S ST RT are projected with J W Σ WT JT; Gaussians are sorted by depth and alpha-composited front to back with alpha = opacity · exp(-½ ΔT Σ'-1 Δ). All four unit tests pass.

The streaks on the thin legs likely come from W: I use camera.R, as the unit test expects, but PyTorch3D stores R for row vectors, so the world-to-camera rotation for column vectors is RT.

part 5
Pre-trained chair: RGB / depth / silhouette

B.6 Training 3D Gaussian Representations

Isotropic Gaussians initialized from the truck point cloud; means, scales, opacities and colors trained with Adam and an L1 loss for 1,000 iterations.

ParameterLearning rate
Opacities (pre-activation)0.05
Scales (pre-activation)0.005
Colors0.02
Means0.0005

Test PSNR 29.48, SSIM 0.938.

Training progress (top: render, bottom: ground truth)
Final renders

B.7.1 Rendering Using Spherical Harmonics

Degree-3 SH (16 coefficients per channel), evaluated with the camera-to-Gaussian direction; color = SH + 0.5, clamped at 0.

DC color only (Q5)
Full spherical harmonics
Frame 0: DC only (left) vs SH (right)

With SH the backrest is a deeper green and its floral pattern is clearer; the DC color, a direction average, looks flatter and lighter.

Frame 8

The seat and backrest are again darker with SH, and the white frame is slightly greyer.

Frame 24

Almost no difference: the matte beige underside has small higher-order SH terms.

B.7.2 Training on a Harder Scene

Materials, validation splitIterationsPSNRSSIM
Baseline (Q6 setup, random init)100018.490.682
Improved200022.950.848
Baseline training progress
Improved training progress
Baseline final renders
Improved final renders

Improved learning rates:

ParameterLearning rate
Opacities (pre-activation)0.05
Scales (pre-activation)0.005
Colors0.02
Means0.002, decayed exponentially to 0.00002
Quaternions0.001

Modifications compared to Q6:

+4.5 dB PSNR and +0.17 SSIM. The baseline leaves the outer spheres as glowing blobs, since all initial points start near the center; the improved run recovers every sphere. Edges stay soft and reflections are missing, because each Gaussian has a single RGB color.