16-825 Assignment 3: Neural Volume Rendering and Gaussian Splatting

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

xy_grid
rays

1.4 Point sampling

1.5 Volume rendering

Color
Depth (normalized by its maximum)

2. Optimizing a basic implicit volume

xyz
Box center0.250.250.00
Box side lengths2.001.501.50

3. Optimizing a Neural Radiance Field (NeRF)

The MLP takes the harmonic embedding of xyz (6 frequencies), has 6 layers with 128 units, and re-injects the embedding at layer 3. Density is a ReLU of a linear head; color is a sigmoid head. Trained for 250 epochs on 128×128 images with 128 stratified samples per ray. Validation PSNR is 24.19 (epoch-200 checkpoint, 100 views).

4. NeRF Extras

4.1 View dependence

The ray direction is embedded with 2 harmonic frequencies and concatenated with the position features after the density head, so only color can depend on view.

With view dependence, 128×128
Without view dependence, 128×128
With view dependence, 400×400
Materials scene (epoch 200)Train PSNRVal PSNRGap
Without view dependence, 128px23.2823.220.06
With view dependence, 128px24.1924.050.14
With view dependence, 400px24.3524.210.14

Left: with view dependence, right: without (same frames). The view-dependent model has slightly brighter highlights on the shiny spheres (mean pixel difference 1.3–1.5/255). Without it, the network can only bake one average color per point, so highlights are dull.

Trade-off. View dependence gave +0.9 dB on both train and validation views. The train/val gap grew only from 0.06 to 0.14 dB, so at this resolution and training length it did not overfit. The risk is that a color that can change freely with direction can also explain wrong geometry: the network can paint each training view separately instead of finding the right surface. That gives good training images but floaters and wrong colors from new viewpoints. NeRF limits this in two ways, which we follow: density depends on position only, and the direction gets a low-frequency encoding (2 frequencies) and enters only the last small color layers. More frequencies or feeding direction earlier would fit specularities better but generalize worse.

4.2 Coarse/fine sampling

A coarse NeRF is evaluated at 64 stratified samples per ray. Its weights wi = Ti(1 − e−σiδi) are normalized into a piecewise-constant PDF along the ray, and 128 more depths are drawn from it by inverse-CDF sampling. A second (fine) NeRF is evaluated on all 192 sorted samples. Both outputs get an MSE loss; the image comes from the fine network. Config: nerf_lego_coarse_fine.yaml.

Lego, 250 epochsSamples per rayTraining speedTraining timeRender timeVal PSNR
Single NeRF (Part 3)128 uniform76 it/s~7 min76 ms/image24.19
Coarse + fine64 + 128 importance35 it/s12 min151 ms/image24.42

Render times are for one 128×128 image, measured back to back on the same GPU; PSNR is over 100 validation views with the epoch-200 checkpoints.

Left: single NeRF, right: coarse + fine. The fine pass puts most samples near the surface, so thin parts such as the tracks and the bucket edge are slightly sharper.

Trade-off. Coarse/fine raises validation PSNR by 0.23 dB, but each step evaluates two networks on 256 points instead of one on 128, so training and rendering are about 2× slower and the model has twice the parameters. Most of the uniform samples in the single-NeRF setup land in empty space; importance sampling spends the budget where the weights are, which matters more at higher resolutions and for thin structures.

4.3 VolSDF

The distance MLP has 6 layers with 128 units on the harmonic embedding of xyz (6 frequencies); a linear head gives the distance. To start from a sensible surface, the output is added to |x| − 1, so the SDF is initialized as a unit sphere and the network learns the residual. Without this, the black background lets the network put black surfaces anywhere that is only seen as background, and the extracted mesh had a large spurious floor. A 2-layer color MLP with a sigmoid takes the same features. The eikonal loss is mean((‖∇d‖ − 1)2) on random points in [−4, 4]3 with weight 0.02. Density is

σ(x) = α Ψβ(−d(x)), where Ψβ is the CDF of a zero-mean Laplace distribution with scale β.

Color, β = 0.01
Geometry, β = 0.01

α and β. α is the density inside the object: far inside, σ → α, so it sets how opaque the solid is. β sets how sharp the transition is: density goes from α to 0 over a band of width about β around the zero level set, and it is exactly α/2 on the surface. As β → 0 the density becomes α times the indicator of the inside.

β (α = 10)Final lossTrain PSNRVal PSNR
0.010.004421.4321.35
0.050.006321.3621.23
0.20.008420.5220.38

Rows: β = 0.01, 0.05, 0.2. Two color frames, then two geometry frames.

  1. How does high β bias the SDF? Low β? With high β the density is spread over a thick shell, and a ray already becomes opaque before it reaches the zero level set. The network can match the images with a smooth, inflated surface, so the SDF is biased towards blobby shapes without thin parts: at β = 0.05 the edges of the base plate are already rounded and swollen, and at β = 0.2 the whole shape is blobby and the bucket floats separately from the arm. With low β the density is close to a step, the ray stops at the zero level set, and the SDF has little bias, so thin parts come out sharp (β = 0.01).
  2. Easier to train with low or high β? High β is easier in general. The density is non-zero over a wide band, so many samples along each ray get gradients even when the surface is in the wrong place, and the result does not depend much on the sample spacing. With low β, only samples very close to the surface get gradients, and if β is smaller than the sample spacing (here 4/128 ≈ 0.03) the surface can fall between samples. In our runs β = 0.01 still trained well, because the sphere initialization starts the surface in a reasonable place and stratified jitter moves the samples every step.
  3. More accurate surface with high or low β? Low β. The rendered surface then coincides with the zero level set of the SDF, so getting the images right means getting the surface right. With high β the images can be matched by many smooth surfaces, so the zero level set is not pinned to the true geometry.

Settings. α = 10, β = 0.01, eikonal weight 0.02, 128 samples per ray, 250 epochs, lr 5e-4 (same schedule as NeRF), and the sphere initialization above. Among the three β values, 0.01 had the lowest loss, the highest validation PSNR and the sharpest geometry (flat base plate, separate tracks, attached bucket). The sphere initialization is what makes such a small β work: the surface starts near the object, so the narrow density band still sees gradients, and eikonal regularization keeps the distance field well-behaved away from the surface.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

6. Training 3D Gaussian Representations

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.02
Means0.0005

Adam, L1 loss, isotropic Gaussians initialized from the point cloud, 1000 iterations. On the held-out views: PSNR 29.22, SSIM 0.936. The training loss went from 0.046 to 0.007.

Training progress (top: render, bottom: ground truth)
Final renders

7. Extensions

7.1 Rendering using spherical harmonics

7.1: with spherical harmonics (degree 3)
5.5: DC term only

In each comparison: left is DC only (5.5), right is full spherical harmonics (7.1), same view.

Frame 4 (top view of the seat and back). With SH the fabric is a darker, more saturated green and the gold pattern stands out more. With DC only, the fabric is lighter and flatter, because the DC term is the color averaged over all viewing directions.

Frame 12. The same effect, more strongly: the back and seat are darker and the gold embroidery has more contrast, while the DC-only render looks washed out. The mean pixel difference is 6.1/255, the largest of the three views (frame 4: 5.3/255).

Frame 20 (underside). This view is mostly the matte wooden base, which looks the same from every direction, so the two renders are almost identical (mean difference 1.8/255). Only the fabric on the side is slightly darker with SH.