16-825: Assignment 3: Volume Rendering, NeRF and 3D Gaussian Splatting

Abhikhya Tripathy (Andrew ID: abhikhyt)

Part A: neural volume rendering. Part B: 3D Gaussian splatting.

1. Differentiable Volume Rendering

1.3 Ray sampling

Each pixel is mapped to the range [-1, 1], placed on the image plane at depth 1, and unprojected to world space. Each ray starts at the camera center and points through its pixel; the direction is normalized.

Pixel grid
Pixel grid (xy_grid)
Ray directions
Ray directions

1.4 Point sampling

64 depths are spaced evenly between near and far. Each sample point is origin + depth × direction.

Sample points
Sample points from the first camera

1.5 Volume rendering

For each sample, alpha = 1 − exp(−σ·Δt). Transmittance T is the product of (1 − alpha) over all earlier samples, with T = 1 for the first one. Each sample's weight is T · alpha. Color is the weighted sum of sample colors, and depth is the weighted sum of sample depths.

Box render
Rendered box (part_1.gif)
Depth map
Depth map, normalized by its maximum

2. Optimizing a Basic Implicit Volume

2.1 Each training step uses a random subset of pixels: all pixel coordinates are shuffled and the first N are kept. This saves GPU memory.
2.2 The loss is the mean squared error between rendered and ground-truth colors.

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Optimized box
Optimized box (part_2.gif). It matches the TA result.

3. Optimizing a Neural Radiance Field

Network: the 3D position is positionally encoded with 6 frequencies, then passed through a 6-layer MLP (128 units, ReLU). The input is re-injected at layer 3 (a skip connection). Two heads follow:

Training: MSE loss, 250 epochs on 128×128 lego images.

NeRF lego
NeRF lego (part_3.gif)

4. NeRF Extras (all three done)

4.1 View dependence

How: the viewing direction is encoded with 2 frequencies and added only to the color head, after the position MLP. Density still depends on position only, so the geometry cannot change with viewpoint.

To answer the trade-off questions with numbers, not just by eye, we measure PSNR (higher is better) on held-out views the model never trained on. This shows how well each model generalizes to new viewpoints.

Materials sceneHeld-out PSNR
128×128, no view dependence23.43 dB
128×128, with view dependence24.09 dB
400×400, with view dependence (lr 5e-4)26.03 dB
No view dependence
128×128, no view dependence
With view dependence
128×128, with view dependence
High-res with view dependence
400×400, with view dependence

Trade-off: view dependence lets shiny materials show highlights that move with the camera. With it, held-out PSNR rose by 0.7 dB. The risk is overfitting: a network that leans too much on direction can "paint" each training view separately instead of learning one consistent 3D scene, and new views then look wrong. We limit this in three ways:

Generalization improved rather than dropped, so this amount of view dependence is a good balance.

At 400×400, the default learning rate (1e-4) collapsed to an empty scene (15.38 dB). Most rays hit black background, so the ReLU density head "died" early and output zero density everywhere. A higher learning rate (5e-4) fixed it.

4.2 Coarse/fine sampling

How: two networks.

LegoSamples / rayHeld-out PSNRTraining speed
Single NeRF (Q3)128 uniform24.51 dB≈ 78 it/s
Coarse + fine64 + 12824.92 dB≈ 44 it/s
Coarse/fine NeRF
Coarse + fine NeRF

Trade-off: samples near the surface matter most, and the fine pass puts them there. Quality goes up (+0.4 dB, sharper edges), but each step runs two networks on more points, so training is about 1.8× slower. The two runs used different GPU nodes, so the speed ratio is approximate.

4.3 VolSDF

How:

What α and β do:

Questions:

  1. How does β bias the SDF? High β spreads density over a thick shell, so the learned surface is smooth and blobby and loses thin parts and sharp edges. Low β puts density in a thin shell, so the surface hugs the true shape closely.
  2. Easier to train with low or high β? High β. Its density is spread out, so many samples along a ray get useful gradients, even when the surface starts in the wrong place. With low β, only samples right at the surface get any gradient, so a poor initial guess gets almost no learning signal. Our low-β run without a good initialization never formed the object: 13.11 dB, renders mostly black.
  3. More accurate surface with low or high β? Low β, provided it trains. The rendering then depends on a thin layer at the true surface, so the SDF's zero level set must sit exactly on the object. With high β, the image can look fine even when the surface is off.
βNo sphere initWith sphere initGeometry with sphere init
0.0113.11 dB (no object formed: renders stayed mostly black with a few scattered blobs)21.61 dBSharpest: flat base, crisp edges
0.0522.06 dB (stray "floor" surfaces)21.64 dBClean, slightly smoother
0.220.73 dB20.57 dBOver-smoothed; thin parts break
VolSDF color
Best: color, β = 0.01 with sphere init
VolSDF geometry
Best: geometry, β = 0.01 with sphere init
High beta geometry
β = 0.2: smoother, loses detail
No init geometry
β = 0.05 without sphere init: stray surfaces

Chosen settings and why: α = 10, β = 0.01, eikonal weight 0.02, 250 epochs, plus sphere initialization. The network predicts a correction on top of the SDF of a radius-1 sphere, so training starts from a valid closed surface instead of noise. This fixes the main weakness of low β (no gradient when the surface starts in the wrong place). Low β can then deliver what it promises, the most accurate surface. It also removed the stray floor surfaces.

5. 3D Gaussian Rasterization

Chair render
Pre-trained chair: color, depth and mask

6. Training 3D Gaussian Representations

Isotropic Gaussians are initialized from the truck point cloud and trained with an L1 loss and Adam, using a separate learning rate for each parameter. Means get the smallest learning rate because small moves in position change the image a lot.

ParameterLearning rate
Opacities0.05
Scales0.005
Colours0.02
Means0.0005
IterationsPSNRSSIM
100029.480.936
3000 (reported)30.780.948
Training progress
Training progress (top: render, bottom: ground truth)
Final renders
Final renders of the training views

7. Extensions (both done)

7.1 Rendering with spherical harmonics

How: for each Gaussian we compute the direction from the camera center to its mean, then evaluate all 16 spherical-harmonic coefficients per color channel (degree 3) in that direction. This gives a color that depends on the viewing angle. Before, only the first, constant coefficient was used.

DC only
Q5.5: constant colour only
With SH
Q7.1: full spherical harmonics

Side-by-side comparisons of the same views. Left: constant colour. Right: full spherical harmonics.

View 16
View 16 (front): with SH the backrest fabric is darker and the gold embroidery brighter and higher-contrast, with visible sheen. Constant colour looks flat and evenly lit.
View 4
View 4 (top-down): SH adds shading and highlights on the seat fabric and gold trim, so the chair looks more 3D. Constant colour looks washed out.
View 24
View 24 (underneath): almost no difference. This view was probably rarely seen during the original training, so the extra SH terms carry little information here.

7.2 Training on a harder scene (materials, random init)

RunPSNRSSIM
Baseline: Q6 setup, random init18.160.680
+ anisotropic, SSIM loss, longer training, LR decay18.700.713
+ wider random init23.080.853
+ wider init + densification23.730.880
+ init matched to scene bounds + densification (best)23.810.884
Baseline progress
Baseline: training progress
Best progress
Best: training progress
Baseline final
Baseline: final renders
Best final
Best: final renders

Changes from the Q6 setup (best run):

  1. Wider initialization (largest gain, ≈ +4.4 dB). The default random init puts every Gaussian in one small ball (std 0.2) at the center, but the materials objects are spread about 2 units wide. Spreading the points to match the scene (std 0.65, 0.55, 0.12 on x, y, z) lets Gaussians start near every object. We estimated these bounds from the density of our 4.1 NeRF. Using a plain std of 0.6 with no scene knowledge gets almost the same result: 23.73 / 0.880.
  2. Densification and pruning. Every 200 steps, for the first 60% of training:
    • Gaussians with opacity below 0.005 are removed.
    • The 10% with the largest position gradients are split into two smaller Gaussians.
    • The total is capped at 25k Gaussians.
    This adds detail where the image error is highest.
  3. Anisotropic Gaussians: rotations are learned too, so Gaussians can stretch to fit edges and flat surfaces.
  4. Loss: 0.8 · L1 + 0.2 · (1 − SSIM), as in the original paper. This favours sharp structure over blur.
  5. Learning rates: means start at 1e-3 and decay exponentially to 1e-5, so they move fast early and settle later.
  6. Longer training: 4000 iterations instead of 1000.

Without the wider initialization, the other changes gave only about +0.5 dB, even with 2× the Gaussians. The narrow initialization was the main bottleneck.