3D Vision Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Q1: Differentiable Volume Rendering

1.3. Visualizing get_pixels_from_image() and get_rays_from_pixels()

Visualized Grid
Visualized Rays

1.4. Point sampling

Sampled Points 1
Sampled Points 2

My computer had some trouble trying to render all points, but here are the beginning and ending rays' samples.


1.5. Volume rendering

Rendered Box
Depth Map

Q2: Optimizing a basic implicit volume

Box Center: (0.2502323091030121, 0.2505885660648346, -0.00048121518921107054)
Box Side Lengths: (2.005110263824463, 1.503585696220398, 1.5033208131790161)

Rendered Box

Q3: Optimizing a Neural Radiance Field (NeRF)

Rendered Lego (High Resolution)

Q4: NeRF Extras

4.1. View Dependence

Rendered Materials (Position Only)
Rendered Materials (View Dependence)

In order to implement view dependence, I separated density and color into separate predictions. While position features went into the regular MLP, color had a separate MLP that processed both the position and direction embeddings. View dependence allows for rendering of different scene elements such as lighting and reflections from different viewpoints; however, there is a risk of overfitting to the training viewpoints and producing colors that look incorrect from novel views.

Q5: 3D Gaussian Rasterization

Rendered Splat

Q6: Training 3D Gaussian Representations

Rendered Splat

Learning rates:
Means: 0.001
Scales: 0.005
Colors: 0.05
Opacities: 0.01
Adam: 0.01

Iterations: 1000
PSNR: 28.234
SSIM: 0.924
Gaussians per splat: 50

Q7: Extensions

Rendered Splat

Learning rates:
Means: 0.003
Scales: 0.005
Colors: 0.002
Opacities: 0.001
Adam: 0.01

Iterations: 300
PSNR: 13.375
SSIM: 0.529
Gaussians per splat: 50

I tried adjusting the learning rates of each parameter to see how they would affect the quality. Due to the intensity of the program, I reduced the iterations and number of Gaussians per splat to 300 and 50, respectively. Unfortunately, this produced an incorrect render, possibly due to not training long enough with low learning rates. I also observed lower PSNR and SSIM values. This might be improved with SSIM loss instead of the MSE loss I had used in part 6 and a higher number of iterations.