Assignment 3

1. Differentiable volume rendering

1.3 Ray sampling

grid
xy grid (mine)
grid TA
xy grid (TA)
rays
rays (mine)
rays TA
rays (TA)

1.4 Point sampling

sample points
sample points (mine)
sample points TA
sample points (TA)

1.5 Volume rendering

spiral
spiral rendering
depth
depth, normalized by max

2. Optimizing a basic implicit volume

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
before
before training
after
after training
gif
spiral after training

3. Neural Radiance Field

NeRF lego
lego (mine)
NeRF lego TA
lego (TA)

Validation PSNR: 23.82 dB

4. NeRF extras

4.1 View dependence (materials)

Density depends on position only; color also takes the encoded view direction.

no view dependence
no view dependence
view dependence
with view dependence
Val PSNR
No view dependence24.10 dB
View dependence25.03 dB

4.2 Coarse / fine sampling (lego)

64 coarse samples, then 64 extra samples drawn from the coarse weights; the fine network renders the merged 128 points. Loss = coarse MSE + fine MSE.

coarse fine
coarse + fine
baseline
single network (part 3)
Single netCoarse + fine
Val PSNR23.82 dB24.02 dB
Render one 128×128 image88 ms228 ms (2.6×)
Parameters93 k186 k

4.3 VolSDF (lego)

Density = α · Ψβ(−d), with Ψβ the CDF of a zero-mean Laplace distribution. Loss = color MSE + 0.02 · eikonal.

VolSDF color
color (β = 0.05)
VolSDF geometry
geometry (β = 0.05)
VolSDF geometry TA
geometry (TA)

What α and β do

Varying β

Top row: geometry. Bottom row: color.

geometry beta 0.01
β = 0.01
geometry beta 0.05
β = 0.05
geometry beta 0.5
β = 0.5
color beta 0.01
β = 0.01
color beta 0.05
β = 0.05
color beta 0.5
β = 0.5
βVal PSNRGeometry
0.0121.47 dBsharpest; cab pillars and base detail kept
0.0521.27 dBgood, slightly smoother
0.518.48 dBblobby; thin structures lost
Best setting: α = 10, β = 0.01, eikonal weight 0.02, sphere initialization of the SDF (radius 1).

Questions

  1. How does β bias the SDF? High β spreads the density over a wide band around the surface, so the zero level set is not pinned down: the SDF becomes smooth and rounded and details wash out. Low β makes the density an almost sharp step at the surface, which keeps details.
  2. Easier to train: low or high β? In principle high β: density is non-zero far from the surface, so rays give gradients that can move the surface. With low β only samples inside the thin band give signal. In my runs all three trained, because the sphere initialization already put the surface close.
  3. Accurate surface: low or high β? Low β: density concentrates at the zero level set, so the extracted mesh is sharper and more faithful (best at β = 0.01, worst at β = 0.5).

5. 3D Gaussian rasterization

chair render
chair: RGB | depth | silhouette

6. Training 3D Gaussians

SettingValue
Learning ratesopacities 0.05 · colours 0.02 · scales 0.005 · means 0.0005
Iterations1000
Test PSNR / SSIM29.25 dB / 0.939
training progress
training progress (top: render, bottom: ground truth)
final renders
final renders of the training views

7. Extensions

7.1 Spherical harmonics

without SH
without SH (5.5)
with SH
with SH (7.1)

Same view, left: without SH, right: with SH.

frame 14
frame 14
frame 1
frame 1
frame 31
frame 31

7.2 Training on a harder scene

Materials scene, random point initialization (no clean point cloud to start from).

baseline training progress
baseline: training progress
improved training progress
improved: training progress
baseline final renders
baseline: final renders
improved final renders
improved: final renders
PSNRSSIM
Baseline (6.2 recipe, isotropic, random init)18.411 dB0.676
Improved19.740 dB0.746

Changes in the improved recipe (vs. 6.2):

Together these give +1.33 dB PSNR and +0.07 SSIM over the baseline on this harder, randomly-initialized multi-object scene.