Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

1. Differentiable volume rendering

Pixels span [-1, 1]. Rays start at the camera center and point toward unprojected image-plane points. Rendering sums color and depth with alpha-transmittance weights.

1.3 Ray sampling

Pixel grid Ray directions

1.4 Point sampling

Samples along rays from the first camera

1.5 Color and normalized depth

Rendered box Depth divided by its maximum

2. Optimizing a basic implicit volume

Center: (0.25, 0.25, 0.00). Side lengths: (2.01, 1.50, 1.50). Adam, learning rate 0.0005, 1,000 epochs, RGB mean squared error. The fitted box matches the target position and dimensions.

Optimized box

3. Neural radiance field

Six 128-unit layers with a skip connection, six position-encoding frequencies, ReLU density, and sigmoid RGB. Trained for 250 epochs on 128 × 128 Lego images, using 1,024 rays and 128 samples per ray.

Lego NeRF spiral rendering
ExperimentPSNR (dB)SSIM
Lego, validation views29.440.942

4. NeRF extensions

4.1 View dependence

Direction encoding (two frequencies) enters only the color branch. Both materials models use 150 epochs, 512 rays, 64 samples, and learning rate 0.0001.

View-independent materials, followed by view-dependent materials:

View-independent materials View-dependent materials
ExperimentPSNR (dB)SSIM
View independent, validation25.080.864
View dependent, validation25.730.871

Adding viewing direction increased validation PSNR by 0.66 dB in this run. View dependence can represent changing highlights. Greater directional capacity can memorize training views; keeping direction out of density and using a small color branch limits that risk.

4.2 Coarse/fine sampling

The coarse network evaluates 64 uniform samples. Its detached weights define a PDF for 64 additional samples; a separate fine network evaluates the 128 sorted samples. Both color losses train the networks. NeRF paper.

Coarse/fine Lego rendering
ExperimentPSNR (dB)SSIM
Uniform (250 epochs, 1,024 rays)29.440.942
Coarse/fine (150 epochs, 512 rays)28.680.932

Median GPU time on the A10G for 1,024 rays (20 forwards after five warm-ups): uniform 5.77 ms; coarse/fine 8.77 ms. Importance sampling concentrates work near visible surfaces, but adds a network pass and can miss features absent from the coarse PDF. These quality results use different training budgets, so they do not isolate the effect of sampling.

4.3 VolSDF

A positional MLP predicts a signed-distance residual to an initial sphere; a second MLP predicts color. The Eikonal loss penalizes (|gradient SDF| − 1)². VolSDF paper.

ExperimentPSNR (dB)SSIM
Beta 0.05, 250 epochs24.400.859
Beta 0.02, 50 further epochs24.810.882

Alpha is 10, the Eikonal weight is 0.02, the initial learning rate is 0.0005, and there are 128 samples per ray. Alpha keeps interiors opaque while the Eikonal penalty regularizes distance gradients. The lower-beta run continues the default run; it is a refinement experiment rather than a controlled beta-only comparison. The displayed model is the lower-beta refinement, selected by validation PSNR.

Best VolSDF color rendering Best VolSDF extracted zero-level surface

Alpha and beta: Alpha scales density and opacity; beta controls the thickness of the transition around the zero-level surface.

High versus low beta: High beta spreads density across a wider region and favors smoother, less precise boundaries. Low beta concentrates density near the surface and permits sharper boundaries.

Easier to train: High beta usually gives useful gradients over a wider region, making coarse sampling and early optimization more forgiving.

More accurate surface: Low beta can give a tighter correspondence between rendered boundaries and the SDF surface, provided sampling resolves the thin transition.

5. Gaussian rasterization

All four supplied Gaussian unit tests pass. The renderer projects covariances, filters and sorts by depth, and blends RGB, depth, and silhouette with front-to-back transmittance. 3D Gaussian Splatting paper. Chair rendering resolution: 128 × 128; displayed panels are enlarged to 256 × 256.

Chair RGB, depth, and silhouette without spherical harmonics

6. Training Gaussian representations

Truck: 10,000 isotropic Gaussians initialized from the provided points, Adam, L1 loss, 1,000 iterations. Learning rates: means 0.0002; scales 0.005; colors 0.01; opacities 0.05. Rotations are fixed for isotropic Gaussians.

ExperimentPSNR (dB)SSIM
Truck, held-out test views29.500.939
Training progress: predictions above ground truth Final training views

7. Gaussian extensions

7.1 Spherical harmonics

All 16 real spherical-harmonic basis functions through degree three modulate RGB using normalized camera-to-Gaussian directions. SH coefficient convention.

Without spherical harmonics:

Chair without spherical harmonics

With spherical harmonics:

Chair with spherical harmonics

Left: DC only. Right: spherical harmonics. Each pair uses exactly the same camera.

Chair frame 8 comparison

Frame 8: spherical harmonics darken the seat upholstery and increase its contrast.

Chair frame 16 comparison

Frame 16: the seat becomes darker, making its worn patches more distinct.

Chair frame 24 comparison

Frame 24: the back view changes less, with only subtle upholstery-color changes. Geometry stays unchanged.

7.2 Harder scene

Baseline: 10,000 randomly initialized isotropic Gaussians, the same learning rates and L1 loss as the truck, and 1,000 iterations.

Improved: 10,000 anisotropic Gaussians initialized uniformly in [-1.2, 1.2]³, initial scale 0.025, and initial opacity sigmoid(-2.2). Mean learning rate starts at 0.001; other rates match the baseline, with rotation rate 0.001. All rates decay exponentially to one tenth over 3,000 iterations. L1 loss is unchanged.

ExperimentPSNR (dB)SSIM
Materials baseline, validation19.240.681
Materials improved, validation26.180.893

The improved setup increased validation PSNR by 6.94 dB. The baseline forms central blobs; the improved model recovers separated objects, although highlights remain soft. Wider initialization covers the separated objects; lower opacity reduces early occlusion. Anisotropic scales fit directional structure, while the longer run and learning-rate decay allow finer adjustment.

Baseline progress and final views:

Training progress: predictions above ground truth Final training views

Improved progress and final views:

Training progress: predictions above ground truth Final training views