16-825 Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

Andrew ID: robertre

A. Neural Volume Rendering

1.3 Ray Sampling

Left: xy grid. Right: ray directions.

1.4 Point Sampling

1.5 Volume Rendering

Left: spiral rendering. Right: depth map, normalized by its maximum.

2.2 Loss and Training

xyz
Box center0.250.250.00
Box side lengths2.011.501.50

2.3 Visualization

3 Optimizing a Neural Radiance Field (NeRF)

4.1 View Dependence

There exists a tradeoff between view-dependence and generalization. A model with view dependence allows the color of a point to change as the viewing angle changes, which is good for capturing reflections and other challenging details. If view dependence doesn't exist in a network, then the color would be an average taken from all the different viewing angles. However, by adding view dependence to the model, the model could focus on getting per view effects right while slightly worsening the underlying geometry to make the training views look good, which could hurt the novel views. In this case, I make the density independent of the viewing angle because density is the underlying geometry, while making the colors change based on the views.

4.2 Coarse/Fine Sampling

Left: one network, 128 samples per ray. Right: coarse network with 64 samples, then fine network with the 64 coarse + 128 new samples.

Compared to the one-pass network, which only does 128 evaluations per ray, the coarse + fine network requires 64 evaluations for the coarse pass and 64+128=192 evaluations for the second pass, which totals to 256 evaluations. In practice, I measured ~65 it/s for the one-pass network and ~29 it/s for the coarse + fine network, with more memory usage as well. However, the coarse + fine network allows us to better recover fine details by allocating evaluation points more intelligently thanks to the initial coarse stage. However, the gain in quality is a bit difficult to tell at 128x128 resolution.

4.3 VolSDF

alpha controls the density / opacity when inside an object. The larger the alpha, the faster the ray stops once it crosses the surface. beta controls how confident we are about the surfaces. The higher the beta, the larger the variance is, meaning we are less certain about the true location of the surface.

  1. How does high beta bias your learned SDF? What about low beta? Answer: Since beta controls how spread out the density is around the surface, a high beta will lead to a smooth surface with higher uncertainty, while low beta tends to give sharper surfaces.
  2. Would an SDF be easier to train with volume rendering and low beta or high beta? Why? Answer: Having a high beta makes the model easier to train as there are more samples along each ray that will get a gradient signal.
  3. Would you be more likely to learn an accurate surface with high beta or low beta? Why? Answer: Lower beta would give a more accurate surface. Since the density is less spread out around the surface, we are more likely to converge to the correct surface.

I used: alpha = 10, beta = 0.05, eikonal weight = 0.02. The model is trained for 250 epochs with a learning rate of 5e-4. I also used geometric initialization (from: https://arxiv.org/abs/1911.10414), so the SDF starts as a sphere of radius 1. With the default initialization, the SDF became negative everywhere and the surface collapsed. The eikonal weight keeps the network close to a valid SDF, and beta = 0.05 is small enough to give a sharp surface.

B. 3D Gaussian Splatting

5 3D Gaussian Rasterization

6 Training 3D Gaussian Representations

ParameterLearning rate
Opacities0.05
Scales0.005
Colours0.005
Means0.0005

Iterations: 1000. PSNR: 29.85. SSIM: 0.944.

Training progress:

Final renders:

7.1 Rendering Using Spherical Harmonics

Without spherical harmonics (5.5):

With spherical harmonics:

Same view, without (left) and with (right) spherical harmonics:

With spherical harmonics, the fabric is darker and the gold pattern stands out more.

With spherical harmonics, the back cushion has a bright gold highlight in the middle and darker sides.

With spherical harmonics, the shadow is less abrupt on the cushion of the chair.

Without spherical harmonics, each Gaussian has one color averaged over all views. With spherical harmonics, the color changes with the view, which adds shading and highlights on the shiny fabric and gold trim.

7.2 Training On a Harder Scene

MethodPSNRSSIM
Baseline (6.2 setup, random init)18.460.678
Improved21.520.752
ParameterBaseline learning rateImproved learning rate
Opacities0.050.05
Scales0.0050.005
Colours0.0050.005
Means0.00050.005
Rotations (quaternions)not trained (isotropic)0.001

Iterations: 1000 for both runs. In the improved run, an exponential schedule multiplies all learning rates by 0.1 over the 1000 iterations.

Baseline training progress:

Improved training progress:

Improved final renders:

With the improved model, the PSNR improved to 21.52 and the SSIM improved to 0.752. I have made three improvements to the training. First, I made the initialization range wider. This is based on the observation from the training progression for the baseline, where Gaussians have to be moved from the center of the scene to cover more objects on the edges. The wider initialization makes the coverage wider to cover more objects earlier in the training. I also increased the learning rate with an exponential decay schedule for the means of the Gaussians so that they can move more freely to model all the different objects in the scene. I also made the Gaussians anisotropic so that they can stretch in each of the 3 dimensions. The 3 improvements allowed the model to capture all objects and higher fidelity for each object.