Assignment 3

1. Differentiable Volume Rendering

1.3. Ray sampling

XY grid visualization Ray bundle visualization
XY grid visualization ground vox grid

1.4. Point sampling

Point samples visualization
Point samples visualization

1.5. Volume rendering

Volume rendering visualization Depth visualization
volume rendering box ground mesh

2. Optimizing a basic implicit volume

2.2. Loss and training

Box center: (0.25023552775382996, 0.250581294298172, -0.00045761160436086357)
Box side lengths: (2.0051209926605225, 1.5035855770111084, 1.5033137798309326)

2.3. Visualization

Optimized volume visualization
 Optimized volume visualization

3. Optimizing a Neural Radiance Field (NeRF)

NeRF visualization
NeRF visualization

4. NeRF Extras

4.1 View Dependence

NeRF visualization
NeRF visualization

5. 3D Gaussian Rasterization

5.5 Perform Splatting

Chair splatting
chair splatting

6 Training 3D Gaussian Representations

Learning rates:

 opacities: 0.05
 scales: 0.005
 colours: 0.001
 means: 0.0001
 quats: 0.05

Iterations: 1000 (default value)
Mean PSNR: 28.906
Mean SSIM: 0.934

Training progress:

Training progress

Final render:

Final render

7.1 Rendering Using Spherical Harmonics

Chair splatting without spherical harmonics
chair splatting without spherical harmonics
Chair splatting with spherical harmonics
chair splatting with spherical harmonics

Comparing frame 10 (top is without spherical harmonics, bottom is with):

frame 10, NO spherical harmonics
frame 10, spherical harmonics used

The green cushions are relatively flat-coloured in the top image where spherical harmonics are not used. In the bottom image where they are used, a realistic gradient appears. This suggests that without view dependency, the colours of gaussians are optimized to be the average of the observed point colours across view directions. For example, if a thin signboard was blue in one hemisphere of direction and red in another hemisphere of direction it would optimize to a purple colour due to lack of expressivity. This suggests how view dependency allows for not only responsive realism, but it also reduces artifacts in situations with challenging view-dependent lighting (such as sun-flares).

Comparing frame 15 (top is without spherical harmonics, bottom is with):

frame 15, NO spherical harmonics
frame 15, spherical harmonics used

Similar to the above observation, we also see how in the bottom image, the shadows on the cushion are more realistic and can convey soft lighting while the shadows in the top image convey a very harsh lighting which seems almost unrealistic.

7.2 Training On a Harder Scene

Baseline

Learning rates (unchanged between baseline and improved):

 opacities: 0.05
 scales: 0.005
 colours: 0.001
 means: 0.0001
 quats: 0.05

Iterations: 1000 (unchanged between baseline and improved)
Mean PSNR : 17.862
Mean SSIM: 0.647

Training progress:

Training progress

Final render:

Final render

Improved

Mean PSNR : 22.699 (up from 17.862)
Mean SSIM: 0.822 (up from 0.647)

Training progress (improved):

Training progress (improved)

Final render (improved):

Final render (improved)

Improvements made:

  1. The gaussians were initialized in a flat grid cuboid as opposed to a random sphere.
  2. The gaussians were made to be anisotropic instead of isotropic.