A. Neural Volume Rendering

1.3. Ray sampling (5 points)

Grid
Rays

1.4. Point sampling (5 points)

Sample points
Sample points

1.5. Volume rendering (20 points)

part_1.gif
Depth

2.1. Random ray sampling (5 points)

2.2. Loss and training (5 points)

Results

Box center: (0.25, 0.25, -0.00)

Box side lengths: (2.01, 1.50, 1.50)

2.3. Visualization

Before (1)
Before (2)
Before (3)
Before (4)
After (1)
After (2)
After (3)
After (4)
Ground Truth
Our Render

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

part_3.gif
part_3.gif

4.1. View Dependence (10 bonus points)

Materials scene, view-dependent NeRF
Materials scene, view-dependent NeRF

View dependence allows rendering of specular surfaces that look different from different angles. Low view dependence will result in a more matte (diffuse) surface, and higher view dependence will allow for sharper glossy (specular) surfaces. Too much view dependence may result in the model “cheating” by generating colors to match just the angles of the input images and not generating a generalized output for other angles, decreasing generalization.

4.2. Coarse/Fine Sampling (10 bonus points)

Coarse: 64 stratified samples per ray. Fine: 128 additional samples from the coarse weights. 192 total.

part_3.gif (stratified, 128 samples)
part_3.gif (stratified, 128 samples)
part_4_2_lego_coarse_fine.gif (coarse/fine)
part_4_2_lego_coarse_fine.gif (coarse/fine)
Frame 5, part_3 (stratified)
Frame 5, part_3 (stratified)
Frame 5, coarse/fine
Frame 5, coarse/fine

Speed decreased due to the need to run sampling twice and increased total number of samples per ray total 128 -> 192. Quality goes up because we waste less samples on inner points that don’t contribute to the render and get more samples on useful points near the surface.

4.3. VolSDF (10 bonus points)

part_4_3_geometry.gif
part_4_3_geometry.gif
part_4_3.gif
part_4_3.gif
Frame 5, geometry
Frame 5, geometry
Frame 5, color
Frame 5, color

Alpha determines how dense the density of material is inside the object.

Beta determines how quickly the density falls off to zero outside the object (sharpness).

  1. How does high beta bias your learned SDF? What about low beta?

    High beta allows material to seep out infront of the surface more, resulting in a fuzzy, fatter object. Lower beta cuts off material closer to the surface, creating a sharper object.

  2. Would an SDF be easier to train with volume rendering and low beta or high beta? Why?

    High beta is easier to train than low beta, The thinness of the surface region impacts how much of the sample provide gradients to the model while learning. Sharper surface region means less samples hit this region and provide gradients.

  3. Would you be more likely to learn an accurate surface with high beta or low beta? Why?

    If both settings are guaranteed to converge, low beta. Once trained, the sharpness and details can be maintained only by the sharper surface of the low beta.

Setting Value
alpha 10.0
beta 0.05
eikonal weight 0.02
epochs 250

Ran a series of 10-epoch tests to empirically find good values.

I think beta follows the sample spacing which is around 0.03 apart. This allows samples to hit the surface region but keep it minimal so allow sharp images. High alpha allows some gradients to pass through so we can accumulate some samples and have a solid surface, but not bleed out into the interior too much.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization (35 points)

part_5_render.gif

6. Training 3D Gaussian Representations (15 points)

Parameter Learning rate
opacities 0.05
scales 0.01
colours 0.05
means 0.001

Iterations: 1000

PSNR: 29.168

SSIM: 0.935

part_6_training_progress.gif
part_6_training_final_renders.gif
part_6_training_final_renders.gif

7.1. Rendering Using Spherical Harmonics (10 points)

Without SH (5.5)
With SH
Frame 6, without SH
Frame 6, with SH

Brighter contrast of green and gold. Deeper shadows in the gold ornamentation.

Frame 16, without SH
Frame 16, with SH

Brighter contrast of green and gold. Darker and sharper shadows of the armrests and backrest.

Frame 26, without SH
Frame 26, with SH

Brighter contrast of green and gold. Darker and sharper shadows around ridges in the back and legs.

7.2. Training On a Harder Scene (10 points)

Baseline

Setup from 6: isotropic Gaussians, random initialization, L1 loss, 1000 iterations.

Parameter Learning rate
opacities 0.05
scales 0.01
colours 0.05
means 0.001

PSNR: 18.866

SSIM: 0.685

part_7_2_training_progress.gif (baseline)
part_7_2_training_final_renders.gif (baseline)
part_7_2_training_final_renders.gif (baseline)

Improved

Baseline Improved
Gaussians isotropic anisotropic
Loss L1 0.8 · L1 + 0.2 · (1 − SSIM)
Mean initialization starter default uniform in [−1.2, 1.2]³
Initial scale / opacity starter default 0.02 / 0.1
Iterations 1000 3000
PSNR 18.866 23.654
SSIM 0.685 0.873
Parameter Learning rate
opacities 0.05
scales 0.005
colours 0.025
means 0.002, decayed exponentially to 0.00002
quaternions 0.001

I’ve based off Claude’s smaller experiments exploring various hyperparams.

This run combines the modifications run on smaller sample runs into one improved attempt:

  • 3000 iterations instead of 1000: more runs.
  • Spread-out initialization: so Gaussians cover all the objects from the start.
  • Low initial opacity, 0.1: Gaussians behind others still get gradients at start of training.
  • Initial scale 0.02: Gaussians cover the scene without starting as fog.
  • Anisotropic Gaussians: they can stretch and rotate to fit edges and highlights.
  • L1 + SSIM loss (0.8 / 0.2): sharper structure.
  • Means learning rate 0.002, decayed to 0.00002: fast moves early, stable settling later.
  • Lower colour and scale learning rates: stable training with the extra per-Gaussian parameters.
part_7_2_training_progress.gif (improved)
part_7_2_training_final_renders.gif (improved)
part_7_2_training_final_renders.gif (improved)
Frame 0, baseline
Frame 0, baseline
Frame 0, improved
Frame 0, improved