Assignment 3: Volume Rendering and Gaussian Splatting

Andrew ID: yunxianm

1. Differentiable volume rendering

I converted the pixel grid to camera NDC coordinates and unprojected it to get world-space rays. The box uses 64 samples per ray between 0 and 5. For each sample, I computed alpha = 1 - exp(-sigma * delta) and multiplied it by the transmittance from earlier samples. Color and depth use these weights.

Pixel grid Ray directions
Pixel grid Ray directions
Sample points from the first camera Depth divided by its maximum
Sample points Depth

Rendered box

2. Optimizing the box

I used RGB MSE and Adam with a learning rate of 0.0005. Each update sampled 4096 rays. After 1000 epochs, the box parameters were:

Parameter x y z
Center 0.25 0.25 0.00
Side lengths 2.01 1.50 1.50

Optimized box

The optimized box looks close to the TA reference, including its proportions and the color pattern around the sides.

3. Neural radiance field

The position MLP has six layers with 128 hidden units. It uses six frequency bands for position encoding and concatenates the input encoding before the fourth layer. Density uses ReLU and color uses sigmoid. I initialized the density output bias to 0.1 to avoid starting with an all-zero density field.

I trained on 100 Lego images for 250 epochs at 128x128, with 1024 rays and 128 samples per ray. The starting learning rate was 0.0005, using the provided decay schedule. Evaluation on the 200 test images gave PSNR 24.43 dB and SSIM 0.8872.

Lego NeRF

4.1. View dependence

For materials, I added a color branch with 64 hidden units and two frequency bands for viewing direction. The density branch still uses only position. Both models used 250 epochs, 128x128 images, and a starting learning rate of 0.0001.

Model Train MSE Test MSE Test PSNR (dB) Test SSIM
Position only 0.00470 0.00509 23.03 0.8469
With direction 0.00400 0.00452 23.57 0.8558

The train MSE was evaluated on the 100 training images after training. It fell by 14.9% with the direction branch. Test MSE fell by 11.2%, and test PSNR increased by 0.54 dB. The improvement on new views shows that the branch is useful, but its larger gain on training images suggests that some of the extra fitting is specific to those views. I expect a larger direction branch could fit more detailed reflections, but I did not test that here.

Materials test comparisons

Position only With direction
Position-only materials model Directional materials model

5. Gaussian rasterization

I projected the 3D covariance to the image plane and composited the visible Gaussians in depth order. The supplied Gaussian tests passed 4/4.

I changed the provided render() to add only the uncovered background, since the RGB sum is already alpha-weighted. I also include the last alpha when passing transmittance to the next chunk.

The animation shows RGB, depth, and silhouette from left to right.

Chair with DC color

6. Optimizing Gaussian representations

I trained 10000 isotropic Gaussians from the supplied Truck point cloud for 1000 iterations, using L1 loss. The position learning rate is small because the point cloud already provides an estimate of the geometry.

Parameter Learning rate
Position 0.0001
Log scale 0.005
Color 0.01
Opacity logit 0.01

On the five test images, mean PSNR was 28.518 dB and mean SSIM was 0.931. The image size was 160x90.

The progress animation has the prediction above the target. The final animation goes through all 197 training views.

Truck training progress

Truck final renderings

Truck test image: target on the left, prediction on the right

Truck test view: target on the left, prediction on the right.

7.1. Spherical harmonics

I used degree-3 spherical harmonics, with 16 basis functions for each color channel. The viewing direction points from the camera center to the Gaussian mean. The C0, C1, C2, and C3 constants and basis ordering follow the reference linked in the assignment.

Chair with spherical harmonics

Frame 4 comparison

At frame 4, SH makes the green seat darker and the pale flower pattern easier to see. The upper-right part of the backrest is also brighter.

Frame 12 comparison

At frame 12, the seat has stronger contrast between the dark green fabric and the light pattern. The backrest also changes in brightness across its surface, while the chair outline stays in the same place.

Frame 20 comparison

At frame 20, the large pale underside looks almost unchanged. The small exposed areas of green fabric show slight brightness differences, so the SH effect is much less visible from this angle.

Running the experiments

Environment: Python 3.11, PyTorch 2.4.1 with CUDA, and PyTorch3D 0.7.9.

Put materials.png and materials.pth from the course archive in nerf/data/. The Lego loader downloads its data automatically.

Run from the code directory:

cd nerf
mkdir -p images ../results/lego_baseline ../results/materials_baseline ../results/materials_view
python volume_rendering_main.py --config-name=box
python volume_rendering_main.py --config-name=train_box
python volume_rendering_main.py --config-name=nerf_lego training.resume=False training.checkpoint_path=checkpoints/lego_baseline.pth training.checkpoint_interval=249 training.render_interval=249 renderer.chunk_size=4096
cp images/part_3.gif ../results/lego_baseline/orbit.gif
python volume_rendering_main.py --config-name=nerf_materials training.resume=False training.checkpoint_path=checkpoints/materials_baseline.pth training.checkpoint_interval=249 training.render_interval=249 renderer.chunk_size=4096
cp images/part_3.gif ../results/materials_baseline/orbit.gif
python volume_rendering_main.py --config-name=nerf_materials +implicit_function.view_dependent=True training.resume=False training.checkpoint_path=checkpoints/materials_view.pth training.checkpoint_interval=249 training.render_interval=249 renderer.chunk_size=4096
cp images/part_3.gif ../results/materials_view/orbit.gif
cd ../GS
python unit_test_gaussians.py
python train.py --out_path output/truck
python render.py --out_path output/chair_dc
python render.py --sh --out_path output/chair_sh
cd ..
python evaluate.py --task nerf --config nerf/configs/nerf_lego.yaml --checkpoint nerf/checkpoints/lego_baseline.pth --output results/lego_baseline
python evaluate.py --task nerf --config nerf/configs/nerf_materials.yaml --checkpoint nerf/checkpoints/materials_baseline.pth --output results/materials_baseline
python evaluate.py --task nerf --config nerf/configs/nerf_materials.yaml --view-dependent --checkpoint nerf/checkpoints/materials_view.pth --output results/materials_view
python evaluate.py --task truck --checkpoint GS/output/truck/gaussians.pth --output results/truck
python make_figures.py --output ../proj3/assets

References