Volume Rendering, NeRF, and 3D Gaussian Splatting

henrytsu · 16-825 · Assignment 3

1. Differentiable Volume Rendering

1.3 Ray sampling

Pixel coordinates Ray directions
Normalized pixel coordinates World-space ray directions

1.4 Point sampling

64 uniformly spaced samples per ray.

Points sampled along the rays

1.5 Volume rendering

20 views at 256 × 256. Depth is normalized by its maximum value.

Spiral rendering Normalized depth
Volume-rendered box Normalized rendered depth

2. Optimizing a Basic Implicit Volume

Optimized parameter Value (rounded to two decimal places)
Center (0.25, 0.25, 0.00)
Side lengths (2.01, 1.50, 1.50)

Spiral rendering of the optimized box

3. Optimizing a Neural Radiance Field

Lego, 128 × 128, 250 epochs. The position MLP has six 128-unit layers, six encoding frequencies, and a skip connection at layer 3. Density uses ReLU; color uses sigmoid.

Setting Value
Rays per iteration / samples per ray 1,024 / 128
Near / far 2.0 / 6.0
Initial learning rate 0.0005
Learning-rate multiplier 0.8^(epoch / 50)
Loss RGB MSE

Lego NeRF spiral rendering

4.1 View Dependence

Viewing direction, encoded with two harmonic frequencies, is added to the color branch. Density depends only on position.

Both materials models use 250 epochs, 128 × 128 images, 1,024 rays per iteration, 128 samples per ray, near/far bounds of 1.0/7.0, and an initial learning rate of 0.0001.

Left: position only. Right: position + viewing direction. Cameras and animation frames match.

Synchronized materials comparison: position only on the left, view dependent on the right

Frame 0 Frame 5
Materials comparison at frame 0 Materials comparison at frame 5

The differences are subtle, mainly in brightness and highlights; reflections remain blurred. View dependence can represent reflections, but its extra capacity can overfit training views. Keeping density independent of direction limits this flexibility. These results are qualitative and do not establish better generalization.

5. 3D Gaussian Rasterization

32 views; RGB, depth, silhouette from left to right.

Gaussian chair renderings: RGB, depth, silhouette

Rotation convention: PyTorch3D's row-vector convention implies W = R.T in the covariance formula. Following the instructor's forum guidance, I use W = R to match the supplied tests. Comparing both conventions showed reduced stretching with R.T at steep angles.

6. Training 3D Gaussian Representations

10,000 isotropic Gaussians initialized from the supplied point cloud; Adam with L1 RGB loss.

Parameter Learning rate
Means 0.00016
Scales 0.005
Colors 0.0025
Opacities 0.05
Setting / metric Value
Iterations 1,000
Resolution / seed 160 × 90 / 1
Held-out PSNR 29.416 dB
Held-out SSIM 0.9399

Training progress

Top: predictions. Bottom: ground truth.

Truck training progress: predictions above ground truth

Final renderings

197 training views; the metrics above use five separate held-out views.

Final trained truck rendered from the training cameras

7.1 Rendering Using Spherical Harmonics

The pretrained coefficients are evaluated using all 16 SH basis functions through degree 3.

Degree-0 color Degree-3 spherical harmonics
Chair rendered with degree-0 color Chair rendered with degree-3 spherical harmonics

Each RGB pair uses the same camera: DC only on the left, full SH on the right.

Frame 0

Matching-view SH comparison at frame 0

SH produces darker green upholstery and stronger contrast in the fabric pattern.

Frame 8

Matching-view SH comparison at frame 8

SH changes local brightness on the fabric and wood. Both retain stretching associated with the covariance convention discussed in Section 5.

Frame 16

Matching-view SH comparison at frame 16

SH gives a less washed-out green surface and a more distinct pattern; the silhouette stays the same.