16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

xy_grid rays

1.4 Point sampling

Sample points along the rays of the first camera (every 16th ray drawn).

1.5 Volume rendering

color depth (normalized by its max)

2. Optimizing a basic implicit volume

2.2 Loss and training

After training, the box has

2.3 Visualization

3. Optimizing a Neural Radiance Field (NeRF)

MLP on harmonic-embedded positions (6 layers, 128 units, skip connection at layer 3), ReLU density and sigmoid color. Trained on lego for 250 epochs at 128×128.

4. NeRF Extras

4.3 VolSDF

geometry color

What alpha and beta do. The density is alpha * Psi_beta(-d), where Psi_beta is the CDF of a zero-mean Laplace distribution with scale beta. Deep inside the object the density is alpha and far outside it is 0, so alpha sets how opaque the solid is. beta sets how wide the transition between the two is: the density falls from alpha to 0 over a band of a few beta around the zero level set. As beta → 0 the density becomes alpha times the indicator of the inside.

I experimented with different betas and measured, over the validation rays that hit the object, how far the rendering weights spread along the ray and how far the rendered depth is from the SDF’s zero crossing on that ray (medians):

beta weight spread along ray rendered depth vs zero crossing validation MSE
0.01 0.096 0.094 0.00650
0.05 0.174 0.082 0.00704
0.2 0.422 0.134 0.00832

  1. High beta spreads the density over a thick shell around the surface, so the rendered color mixes points well in front of/behind the zero level sets. That is why the SDF comes out blobby, with an inflated base and the bucket arm broken off at beta = 0.2. Low beta concentrates the weights at the zero crossing, so the rendering behaves almost like surface rendering of the zero level set and the SDF is pushed onto the real surface.
  2. High beta is easier to train. because the gradient of the density with respect to d is proportional to exp(-|d| / beta). So with a low beta only samples right next to the current surface get any signal and the surface cannot be pulled from far away.
  3. Low beta gives the more accurate surface, for the reason in (1): the density is close to a step at the zero level set, so what the images constrain is the zero level set itself. In the table it has the tightest weights and the lowest validation error.

Settings. alpha = 10, beta = 0.01, eikonal weight 0.02, 250 epochs. The SDF network outputs a residual on top of |x| - 1 (a unit sphere): without it the initial SDF was negative everywhere and stayed so, and no surface was ever extracted. beta = 0.01 was chosen from the sweep above: it has the lowest validation error and the cleanest geometry, and with the sphere initialisation it no longer needs a large beta to get training started.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

6. Training 3D Gaussian Representations

parameter learning rate
opacities (pre-activation) 0.05
scales (log) 0.01
color 0.02
means 0.001

Trained for 3000 iterations with L1 loss.

Held-out views: PSNR 31.068, SSIM 0.952.

Training progress (top: renders, bottom: ground truth):

Final renders of the training views:

7. Extensions

7.1 Rendering Using Spherical Harmonics

DC only (Q5.5) full spherical harmonics

Side-by-side comparisons of the same view (left: DC only, right: spherical harmonics):

With only the DC term every Gaussian has the same color from all directions, so the fabric looks evenly lit and flat. With spherical harmonics the seat and back pick up shading that depends on which way they face the camera: the seat cushion is darker and the brocade pattern stands out more against the fabric.

Seen from the front, the DC render is a uniform bright green. The spherical harmonic render is darker overall (mean brightness of the chair drops from 143 to 125 out of 255) while its brightest pixels get slightly brighter, which shows up as a sheen on the satin stripes and on the gilded carving.

From below, the underside of the seat is the same flat cream color in both, but the green fabric and the gold ornaments on the legs have more contrast with spherical harmonics (the standard deviation of brightness over the chair rises from 40 to 49).