16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

A. Neural Volume Rendering

1.3 Ray sampling

Pixel grid (xy_grid)
Ray directions

1.4 Point sampling

Stratified point samples from the first camera

1.5 Volume rendering

Spiral rendering of the box
Depth, normalized by its maximum

2. Optimizing a basic implicit volume

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Ours: spiral rendering of the optimized box
TA reference

3. Optimizing a Neural Radiance Field (NeRF)

Ground-truth training images, one view per frame
NeRF on lego, 128×128, 250 epochs

Final training loss (MSE, last epoch): 0.002985

4. NeRF Extras

4.1 View dependence

View-dependent NeRF on materials, 400×400 (nerf_materials_highres)
Materials, 128×128, without view dependence
Materials, 128×128, with view dependence
Materials, 128×128Mean PSNRMean SSIMParametersTraining speedTraining time (250 epochs)Inference (128×128 frame)
Without view dependence23.6250.870481,79621.3 it/s19.6 min233 ms (4.3 fps)
With view dependence24.3080.884582,02018.6 it/s22.4 min266 ms (3.8 fps)

We see a clear improvement from using view dependence, both in the metrics and visually. It is most visible on the grey balls in the center, which for me showed the view dependence best. The scene is still fairly low resolution for major differences to show up. I think that with training at a higher resolution, we would have seen a larger gap in both the metrics and the visual quality.

The cost of view dependence is about 100K extra parameters for the direction branch and about 14% slower training and rendering (18.6 vs 21.3 it/s, 266 vs 233 ms per frame). The risk for generalization is that the network can explain the training views with view-dependent colour instead of learning the actual geometry, which would hurt novel views.

4.2 Coarse/fine sampling

Lego, single network
Lego, coarse + fine networks
Materials 400×400 with view dependence, single network
Materials 400×400 with view dependence, coarse + fine networks
RunMean PSNRMean SSIMParametersNetwork evaluations per rayTraining speedTraining time (250 epochs)Inference (per frame)
Lego 128×128, single network25.3070.912481,79612829.5 it/s14.1 min163 ms
Lego 128×128, coarse + fine25.7120.921963,592128 + 256 = 38411.0 it/s38.0 min507 ms
Materials 400×400 + view dependence, single network24.4050.853582,02012820.0 it/s20.9 min2.72 s
Materials 400×400 + view dependence, coarse + fine24.4160.8521,164,040128 + 256 = 3847.6 it/s55.0 min11.70 s

The fine network evaluates the 128 stratified samples plus 128 importance samples drawn from the coarse weights. Each ray therefore costs about 3× the network evaluations of the single-network model, which matches the 2.6–2.7× slower training we measured. Rendering runs both passes as well, so inference is 3.1× slower on lego and 4.3× slower on materials at 400×400.

So we did not get any huge gain in quality overall (+0.4 dB PSNR on lego, essentially none on materials), but the runtime went up a lot, so for these scenes coarse/fine sampling does not seem worth it.

4.3 VolSDF

N/A

B. 3D Gaussian Splatting

5. 3D Gaussian rasterization

Pre-trained chair Gaussians rendered with render.py (RGB, depth, mask)

6. Training 3D Gaussian representations

ParameterLearning rate
Opacities0.05
Scales0.005
Colours0.02
Means0.0005
IterationsMean PSNRMean SSIM
100029.5840.938
Training progress (top: prediction, bottom: ground truth)
Final renders of the training views

7. Extensions

7.1 Rendering using spherical harmonics

Q5.5: DC colour only (RGB, depth, mask)
Q7.1: all spherical harmonic components up to degree 3 (RGB, depth, mask)

Side-by-side comparisons (left: DC only, right: with spherical harmonics)

Frame 3

With spherical harmonics there is a bit more contrast on the objects and the details look much better. The back of the chair has a difference in colour with SH, whereas without it we don't see that difference. It also looks sharper than the render without SH.

Frame 12

Same as frame 3: the back of the chair shows the difference in colour with SH that is missing without it, and it looks sharper with SH.

7.2 Training on a harder scene

MapAnything point cloud used to initialize the Gaussians

The Gaussian means are initialized from a point cloud predicted by MapAnything instead of the random init. The pipeline (GS/mapanything_init.py) has five steps:

  1. It runs MapAnything on 24 full-resolution (800×800) training views, chosen by farthest-point sampling on camera direction. The ground-truth intrinsics and poses are passed in as inputs.
  2. It aligns the predicted point maps to the ground-truth world frame with a similarity transform fitted on the 24 camera centres (scale 1.065, mean centre residual 0.036).
  3. It drops background pixels, pixels MapAnything marks invalid, and the lowest 10% by confidence.
  4. It applies a silhouette-consistency filter, which keeps a point only if it projects onto the foreground in at least 95% of the 100 training views where it is in frame. This step removed 44% of the points.
  5. It merges points per voxel (23,415 points) and then samples 10,000 of them uniformly, keeping their colours, to match the baseline's Gaussian count.
Random init (baseline), before training
MapAnything init, before training (interactive viewer)

Runs

Each run adds one change on top of the previous one. All runs use 10,000 Gaussians, 1000 iterations at 128×128, and the Q6.2 learning rates: opacities 0.05, scales 0.005, colours 0.02, means 0.0005. Quaternions use 0.001 and higher-order spherical harmonics 0.0035 where they are trained.

RunInitGaussiansColourLossMean PSNRMean SSIM
Baseline (Q6.2 setup)randomisotropicRGBL118.4560.683
MapAnything initMapAnything points + coloursisotropicRGBL123.4310.868
+ anisotropicMapAnything points + coloursanisotropicRGBL123.6000.871
+ view dependenceMapAnything points + coloursanisotropicSH degree 3L125.2110.901
+ SSIM loss (best)MapAnything points + coloursanisotropicSH degree 30.8 L1 + 0.2 (1 − SSIM)25.3490.910

Training loss

Training loss, L1 runs
Training loss, + SSIM loss

Turntable comparison

All five runs rendered on the same orbit as the NeRF materials spiral

Summary

The highest gain came from using an external model (MapAnything) for the initialization: PSNR went from 18.456 to 23.431 and SSIM from 0.683 to 0.868, and the loss plateaued a lot earlier. Initializing from this model is easier than starting from a random init and saves training time. With the random init our scene did not even converge in the given iterations.

Anisotropic Gaussians gave a small gain (23.431 to 23.600 PSNR, 0.868 to 0.871 SSIM), since each Gaussian can stretch along the surface instead of being a sphere. View dependence gave the second largest gain (23.600 to 25.211 PSNR, 0.871 to 0.901 SSIM), since the materials are shiny and their colour changes with the viewing direction, which a single RGB colour per Gaussian cannot represent.

Adding the SSIM loss also helped (25.211 to 25.349 PSNR, 0.901 to 0.910 SSIM), but by a smaller margin than the init and view dependence. It is about the same as anisotropic in PSNR, and larger in SSIM since that is the metric it directly optimizes.

Training progress (top: prediction, bottom: ground truth)

Random init (baseline)
MapAnything init
+ anisotropic
+ view dependence
+ SSIM loss (best)