16-825 Learning for 3D Vision

Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

A. Neural Volume Rendering

1Differentiable Volume Rendering

Volume-rendered box from a spiral of viewpoints
Rendered box
Depth map of the box
Depth

2Optimizing a basic implicit volume

Before training
Box before training, view 0
Box before training, view 1
Box before training, view 2
Box before training, view 3
After training
Box after training, view 0
Box after training, view 1
Box after training, view 2
Box after training, view 3
Final result
Optimized box from a spiral of viewpoints

3Optimizing a Neural Radiance Field (NeRF)

NeRF rendering of the lego scene

4.3NeRF Extras: VolSDF

For this section, I chose 4.3.
alpha 20.0
beta 0.05
eikonal_weight 0.05

alpha = 20.0, beta = 0.05, and eikonal_weight=0.05 gives me the result below. beta is chosen as 0.05 because it s about as sharp as possible while the density transition still covers several samples per ray; alpha is chosen as 1/beta because it makes rays opaque within about 0.1 units of the surface, so surfaces look solid and thin parts aren't see-through; eikonal_weight of 0.05 keeps the network a reasonable distance function as it keeps the surface smooth and it's weak enough to preserve the details.

Extracted VolSDF surface geometry of lego
Geometry
VolSDF color rendering of lego
Rendering
  1. High beta makes density spread over a wide region around the surface of the model, which will result the model to match the image with blurry, thick shells. The learned surface will be overly smooth, which results in the loss of details. On the other hand, low beta makes density concentrated right at the zero-level set, which biases the SDF towards a thin, accurate surface. However, small SDF errors can also turn into holes or floating artifacts.
  2. SDF will be easier to train with volume rendering and high beta, because points away from the surface still have non-zero density so they can contribute useful gradients. This makes training easier as the optimization can be stablized even when the starting surface is far off. With low beta, only samples close to the surface matter, and that makes useful gradients sparse.
  3. I am more likely to learn an accurate surface with low beta, because with low beta, the density acts like a sharp step at the point that is on the surface.

B. 3D Gaussian Splatting

53D Gaussian Rasterization

Rendered chair from pre-trained Gaussians: RGB, depth and mask

6Training 3D Gaussian Representations

Mean PSNR 29.304
Mean SSIM 0.937
ParameterLearning rate
Opacities0.05
Scales0.005
Colors0.01
Means1.6e-4

Number of iterations: 1000

Final renders of the trained truck Gaussians
Final renders
Training progress of the truck Gaussians
Training progress

7.1Extensions: Rendering Using Spherical Harmonics

For this section, I chose 7.1.
Chair rendered without spherical harmonics
Part 5 render result (without SH)
Chair rendered with spherical harmonics
Part 7 render result (with SH)

Image comparisons

The depth maps and silhouettes are identical in every pair, because spherical harmonics change only the color of each Gaussian, not its position, shape or opacity. All differences are in the RGB image.

Frame 003
Frame 003 without spherical harmonics
Without SH (Q5.5, DC only)
Frame 003 with spherical harmonics
With SH (Q7.1)

With SH, the fabric is a deeper, more saturated green and the gold floral pattern on the backrest and seat stands out clearly.

Frame 019
Frame 019 without spherical harmonics
Without SH (Q5.5, DC only)
Frame 019 with spherical harmonics
With SH (Q7.1)

This view shows mostly the matte beige underside of the seat, which looks the same from every direction, so the two renders are very similar.

Frame 031
Frame 031 without spherical harmonics
Without SH (Q5.5, DC only)
Frame 031 with spherical harmonics
With SH (Q7.1)

With SH, a gold ornament down the centre of the backrest is clearly visible and the gilded arm supports show bright metallic highlights. Without SH, the backrest looks nearly uniform, and the ornament is barely visible.

Why

Without SH, each Gaussian has one fixed color, so it looks the same from every angle. Real materials don't behave like that: shiny things like the gold thread and the gilded frame look brighter from some angles and darker from others. With only one color, those highlights get averaged away, so the chair looks flatter and more washed out. SH let each Gaussian's color change with the viewing direction, so the shine and contrast come back. Matte parts look the same from every angle, which is why they barely change.