Assignment 3

NeRF & Gaussian Splatting

Krisha Chemburkar
Part A · Neural Volume Rendering

1 Differentiable volume rendering

1.3 Ray sampling

Pixel grid
grid
Rays
rays

1.4 Point sampling

Sample points
sample points

1.5 Volume rendering

Box render
render
Depth map
depth

2 Optimizing a basic implicit volume

Box center
(0.25, 0.25, 0.00)
Side lengths
(2.01, 1.50, 1.50)
Optimized box
part_2.gif

3 Neural Radiance Field

Lego NeRF
part_3.gif

4.1 View dependence

Materials scene, 128 px
128×128
Materials scene, 400 px
400×400

Trade-off. View dependence lets the model get the shiny highlights on the spheres right, which a plain NeRF can't do. But if color depends too much on direction, the network can cheat: instead of learning the right shape, it just changes the color for each training view. That looks fine on training images but breaks on new views. To avoid this I kept density position-only, added the direction late, and used only a couple of frequencies for it.

4.3 VolSDF

VolSDF geometry
geometry
VolSDF color
color

α and β

α is how dense the inside of the object is, so it controls how solid it looks. β controls how sharp the surface is. Small β means density jumps from 0 to α right at the surface. Large β means it fades in slowly, like a fuzzy shell.

  1. How does high β bias the SDF? What about low β? High β makes the surface blurry, so the learned shape ends up smooth and a bit puffy, and thin parts get lost. Low β gives a sharp surface that has to be in exactly the right place.
  2. Easier to train with high or low β? High β. Density is spread out, so lots of points along each ray get gradients even when the surface is in the wrong spot. With low β only points right at the surface get any signal.
  3. More accurate surface with high or low β? Low β. The density sits right on the surface, so what gets rendered matches where the SDF is actually zero.

Settings

I used α = 10, β = 0.05 and an eikonal weight of 0.02. β = 0.05 is small compared to the object, so the surface comes out sharp, but it's still wide enough to train without trouble. α = 10 makes the inside solid after just a few samples. The eikonal weight keeps the SDF valid without drowning out the color loss.

Part B · 3D Gaussian Splatting

5 3D Gaussian rasterization

Gaussian splatting render
render.py output

6 Training 3D Gaussians

Learning ratesIterationsPSNRSSIM
opacities 0.05 · scales 0.005 · colours 0.0025 · means 0.0001 1000 29.25 0.938
Truck training progress
training progress
Truck final renders
final renders

7.1 Rendering using spherical harmonics

Render with spherical harmonics
with spherical harmonics (7.1)
Render without spherical harmonics
without spherical harmonics (5.5)

Side-by-side (left: without SH, right: with SH)

View 5 comparison
View 5: with SH the green fabric has darker, more varied shading and the gold pattern stands out more. Without SH it looks flatter and lighter.
View 12 comparison
View 12: the seat and back pick up lighting differences with SH (darker in places, brighter highlights on the pattern). Without SH every part has one fixed color.
View 20 comparison
View 20: the underside looks almost the same in both, since it's mostly flat beige. The green parts are a bit richer with SH.

7.2 Training on a harder scene

Materials scene with random initialization.

RunIterationsLossMeans LRPSNRSSIM
Baseline (6.2 setup)1000L10.000118.100.658
Improved30000.8·L1 + 0.2·(1 − SSIM)0.01 → 0.000120.920.779

The other learning rates are the same as Q6 in every run (opacities 0.05, scales 0.005, colours 0.0025).

Training progress

Baseline training progress
baseline
Improved training progress
improved

Final renders

Baseline final renders
baseline
Improved final renders
improved (best)

What I changed