16-825 Assignment 3

Volume Rendering, NeRF, and 3D Gaussian Splatting

Name: Yu-Rou Tuan (andrew id: yurout)

A. Neural Volume Rendering

1.3 Ray sampling

xy grid
rays

1.4 Point sampling

sample points from the first camera

1.5 Volume rendering

color
depth (normalized)

2. Optimizing a basic implicit volume

After training:

optimized box

3. NeRF

lego

4.1 View dependence

materials, 128x128
materials, 400x400

The trade-off is that the more the color can depend on direction, the easier it is for the network to cheat: it can fit each training image by changing color per view instead of getting the geometry right, and then novel views look worse. Keeping the direction encoding low frequency and feeding it in late limits how much it can do that.

4.2 Coarse/fine sampling

The coarse network uses 64 evenly spaced samples. Its weights are used as a distribution to draw 128 more samples, mostly near the surface, and the fine network renders with all of them.

single network (part 3)
coarse/fine
Val PSNRTime per step
Single network, 128 samples24.35 dB6.1 ms
Coarse/fine24.47 dB9.1 ms

Coarse/fine is about 1.5x slower since there are two networks and more samples. At this low resolution the quality gain is small. It should matter more at higher resolution, where evenly spaced samples miss thin details.

4.3 VolSDF

geometry
color

alpha and beta. alpha is the density inside the object, so it controls how opaque the surface is. beta controls how quickly density falls off around the surface. Small beta gives a sharp jump at the surface, large beta makes it a blurry, thick shell.

1. How does beta bias the SDF? With high beta, density leaks out in front of the surface, so the rendered surface ends up smoother and slightly off from the true zero level set. With low beta, density is almost a step function, so the surface sits right at the zero level set.

2. Easier to train with low or high beta? High beta. Density is spread out, so many samples along a ray get gradients even when the SDF is still wrong. With low beta only the points very close to the surface get any gradient, which makes it hard to get started.

3. More accurate surface with low or high beta? Low beta. The rendering weights peak right at the zero crossing, so the geometry has to be correct to match the images, while high beta lets a blurry surface still look fine.

Settings. I tried three values of beta with everything else the same (alpha = 10, eikonal weight 0.02).

beta = 0.02
beta = 0.05
beta = 0.2

beta = 0.05 worked best and is the result shown above. With 0.02 the network barely learned anything, since only points right at the surface get gradients. With 0.2 it only learned a blurry blob and no surface could be extracted. In both cases the mesh was empty. I also used Softplus instead of ReLU in the SDF network so the SDF stays smooth, which helps the eikonal loss.

B. 3D Gaussian Splatting

5. Rendering pre-trained Gaussians

color / depth / mask

6. Training Gaussians

ParameterLearning rate
opacities0.05
scales0.005
colours0.02
means0.0005

Trained for 1000 iterations with L1 loss.

PSNR: 29.52   SSIM: 0.939

training progress (top: render, bottom: GT)
final renders

7.1 Spherical harmonics

DC only (part 5)
full spherical harmonics

Left: DC only. Right: full SH. Same view in each pair.

view 3

With SH the green fabric has darker shading and the gold pattern stands out more. DC only looks flatter and a bit washed out because it's the average color over all directions.

view 10

The seat and back are a lot darker with SH, and the pattern on the seat has more contrast. With DC only, every surface gets the same color from every angle.

view 22

From below the two look almost the same. The bottom is mostly flat beige with little view-dependent effect, so SH doesn't add much here.

7.2 Harder scene

Baseline (same setup as part 6: isotropic, random init, 1000 iterations)

PSNR: 18.60   SSIM: 0.681

baseline training progress

Improved

ParameterLearning rate
opacities0.05
scales0.005
colours0.02
means0.002
quaternions0.001

Trained for 5000 iterations with L1 loss.

PSNR: 22.95   SSIM: 0.844

improved training progress
improved final renders

The baseline only ends up with a blurry blob near the middle and most of the spheres on the outside never show up. The improved version gets all the objects and is close to the ground truth, though the edges are still a little soft.

What I changed: