Assignment 3

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

Left: xy grid, right: rays.

1.4 Point sampling

1.5 Volume rendering

Left: color, right: depth.

2. Optimizing a basic implicit volume

Center = (0.25, 0.25, 0.00). Side lengths = (2.00, 1.50, 1.50).

3. Optimizing a Neural Radiance Field (NeRF)

4. NeRF Extras

4.1 View Dependence

The density depends only on position, but color depends on the viewing direction. Since the direction embedding has fewer frequencies, color can end up varying more slowly with view direction.

128x128. Left: baseline, right: view dependent.

400x400. Left: baseline, middle: view dependent, right: larger view dependent model.

Both small models are blurry, and neither can produce a good shiny/polished sphere, presumable because its appearance changes very quickly with viewing direction. The difference between the baseline and the view dependent model is mostly on the shiny spheres, where the view dependent model changes the brightness with the view. The larger view dependent model is much better, and starts to show some of the reflection on the mirror sphere.

With one color per point, the only way to fit all the training images is the correct geometry, so the multi-view consistency of the images forces the geometry to be correct. If the color can change freely with the direction, it becomes easier to obtain wrong geometry when the camera is far from the cameras that supervised training and still have good renders when the supervised cameras are used - overfitting. View dependence does not make the geometry wrong directly, but allows the network to represent bad geometry more easily since it can hack the loss during optimization.

4.3 VolSDF

Alpha is the density inside of the object, and so varying this controls how much light can attenuate along a ray passing through an object. Beta is essentially how "wide" the surface of the object is - how quickly we can transition from being inside the object to outside when traveling normal to its surface.

How does high beta bias your learned SDF? What about low beta?
With high beta, the render can be blurry because the boundary of the object is "stretched" across space, normal to the actual surface. This can also cause thin parts to disappear completely or break. With low beta, the surface is sharper, but this also makes optimization harder since the probably of sampling a point exactly in this region is lower, and so the gradients are more sparse.

Would an SDF be easier to train with volume rendering and low beta or high beta? Why?
Higher beta is easier to optimize, since the chances of getting a sample in the diffuse region of the boundary is more.

Would you be more likely to learn an accurate surface with high beta or low beta? Why?
Lower beta produces more accurate surfaces, though there is still a bias near the object's actual surface. Since the region representing the boundary is both thinner and closer to the actual surface of the object, the representation is more accurate.

Best result:
Eikonal weight = 0.1, alpha = 10, beta = 0.05, num epochs = 250, learning rate = 5e-4, MLP: 6 layers, 128 units, Softplus (beta = 100) (from the IDR paper), 6 frequencies.

The default eikonal weight of 0.02 leaves a large flat surface underneath the model. My hypothesis is that due to there not being enough variation in camera positions to supervise this region below the model, it doesn't end up getting pinned by the color loss. A larger eikonal weight removes this.

Beta. All with eikonal weight 0.1 and alpha = 10.

The best result was with beta = 0.05. I also tried a few other values:
beta = 0.2:
The renderer gets enough density from regions where the SDF comes close to 0 but stays positive. So thin parts can appear in the render with no zero crossing, and marching cubes loses them. Its possible that a different threshold for marching cubes can recover some of this detail, but this affects the whole model and may end up inflating it.

beta = 0.01:
The hypothesis here is as above - with small beta values, there isn't enough signal to actually learn the model.

These were attempted without varying the alpha values. The original paper mentions that they set alpha to the reciprocal of beta, so as to attenuate the light that gets past an object as its boundary becomes more diffuse. Setting a rule for alpha based on beta, and then scheduling beta from coarse to fine may make it easier to learn and produce high quality renders, at the cost of training time.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

Note: when computing the 2D covariance, using camera.R makes the unit tests pass, but using camera.R.transpose(-1, -2) produces better outputs and reconciles there difference between the PyTorch3D and 3DGS conventions (and so I use the transposed version).

6. Training 3D Gaussian Representations

I trained for 1000 iterations. The learning rates are as follows:

PSNR: 30.26, SSIM: 0.946.

7. Extensions

7.1 Rendering Using Spherical Harmonics

DC spherical harmonics (from question 5.5)

All spherical harmonics

Frame 4. Left: DC SH, right: All SH.

Frame 12. Left: DC SH, right: All SH.

Frame 20. Left: DC SH, right: All SH.