16-825 Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting
Andrew ID: tzuyiy
A.1 Differentiable Volume Rendering
1.3 Ray sampling
NDC pixel coordinates in [-1, 1] are unprojected to world space at depth 1; each ray direction is the normalized vector from the camera center to that point.


1.4 Point sampling
Evenly spaced depths between min_depth and max_depth: origin + z · direction.

1.5 Volume rendering
alpha_i = 1 - exp(-sigma_i * delta_i)
T_i = prod_{j < i} (1 - alpha_j)
w_i = T_i * alpha_i
color = sum_i w_i * c_i depth = sum_i w_i * z_i


A.2 Optimizing a Basic Implicit Volume
Trained with an MSE loss on randomly sampled pixels.
| Box center | (0.25, 0.25, 0.00) |
|---|---|
| Box side lengths | (2.01, 1.50, 1.50) |



part_2.gif)A.3 Optimizing a Neural Radiance Field
6 harmonic frequencies, then a 6-layer, 128-wide MLP with a skip connection at layer 3; ReLU density head and sigmoid color head. Default nerf_lego.yaml settings: 250 epochs, 1,024 rays, 128 samples per ray, lr 5e-4.




Validation PSNR 24.55 dB, SSIM 0.893.
A.4.1 View Dependence
The direction (2 harmonic frequencies) is concatenated to the trunk features and fed only to the color head (Linear → 64 → ReLU → 3 → Sigmoid), so density depends on position only. Both models use nerf_materials.yaml and differ only in this switch.
| Materials, 128×128 | Val PSNR | Val SSIM |
|---|---|---|
| No view dependence | 23.26 | 0.847 |
| View dependence | 23.82 | 0.855 |





Trade-offs. View dependence raised PSNR from 23.26 to 23.82 dB. The position-only model bakes whitish blotches into the copper and gold spheres; the view-dependent model is smoother, though neither captures the sharp reflections at 128×128. Too much direction capacity hurts generalization: the network can fit each training view by changing color instead of geometry. I keep it small: 2 frequencies, color head only.
A.4.2 Coarse/Fine Sampling
Two Part 3 MLPs. The coarse weights at 64 evenly spaced samples define a PDF along the ray, from which 128 more samples are drawn by inverse-CDF sampling (weights detached). The fine network renders all 192 sorted samples; loss = coarse MSE + fine MSE. Both models: 25,000 updates.
| Lego, 128×128 | Network evals / ray | Train speed (CPU) | Val PSNR | Val SSIM |
|---|---|---|---|---|
| Single network, 128 samples (Part 3) | 128 | 7.6 it/s | 24.55 | 0.893 |
| Coarse/fine, 64 + 128 samples | 64 + 192 | 3.9 it/s | 24.94 | 0.898 |


Trade-offs. +0.39 dB, because the extra samples go near the surface instead of empty space. The cost is two networks, 256 instead of 128 evaluations per ray, and half the training speed (3.9 vs 7.6 it/s on CPU) for the same number of updates.
A.4.3 VolSDF
A 6-layer, 128-wide MLP predicts a residual on a unit-sphere SDF, d(x) = MLP(x) + ||x|| - 1, plus features; a 2-layer color head ends in a sigmoid. Density is alpha · Psi_beta(-d) (Laplace CDF). Eikonal loss weight 0.02 on random points in [-4, 4]³.
What alpha and beta do. alpha is the maximum density, reached deep inside the object; density is alpha/2 on the surface and 0 far outside. beta is the width of this transition; as beta → 0 density becomes a step at the surface.
- How does high / low beta bias the SDF? High beta makes space outside the surface partly opaque, so the images are fit by a blurred volume and the SDF comes out too smooth: thin parts and edges get rounded (my beta 0.1 result). Low beta has little bias: the visible surface coincides with the zero level set.
- Easier to train with high or low beta? High beta. The density gradient covers a wide band, so samples far from the current surface still guide the SDF. With low beta only samples in a thin band get gradients, and rays can step over it.
- More accurate surface with high or low beta? Low beta, as long as it still trains: only then must the zero level set match the true surface. Here beta 0.02 gave the sharpest geometry and the best PSNR.
Results and hyper-parameters
alpha = 10; other settings from volsdf_surface.yaml.
| Lego, alpha = 10 | Val PSNR | Val SSIM |
|---|---|---|
| beta = 0.1 | 21.20 | 0.787 |
| beta = 0.05 (config default) | 21.43 | 0.798 |
| beta = 0.02 | 21.59 | 0.819 |
Best result: beta = 0.02.





Lower beta gives better metrics and sharper geometry. At 0.1 the base plate is a rounded blob and the arm merges with the body; at 0.02 the edges are sharp and the parts are separated. With the sphere init, small beta trained stably.
Effect of the sphere initialization


Without it, beta 0.05 converged but grew a bumpy false floor in the black background, which the color loss does not penalize, and beta 0.1 and 0.02 diverged (~13 dB). The sphere init fixes both at a cost of about 0.5 dB.
B.5 3D Gaussian Rasterization
Covariances R S ST RT are projected with J W Σ WT JT; Gaussians are sorted by depth and alpha-composited front to back with alpha = opacity · exp(-½ ΔT Σ'-1 Δ). All four unit tests pass.
The streaks on the thin legs likely come from W: I use camera.R, as the unit test expects, but PyTorch3D stores R for row vectors, so the world-to-camera rotation for column vectors is RT.

B.6 Training 3D Gaussian Representations
Isotropic Gaussians initialized from the truck point cloud; means, scales, opacities and colors trained with Adam and an L1 loss for 1,000 iterations.
| Parameter | Learning rate |
|---|---|
| Opacities (pre-activation) | 0.05 |
| Scales (pre-activation) | 0.005 |
| Colors | 0.02 |
| Means | 0.0005 |
Test PSNR 29.48, SSIM 0.938.


B.7.1 Rendering Using Spherical Harmonics
Degree-3 SH (16 coefficients per channel), evaluated with the camera-to-Gaussian direction; color = SH + 0.5, clamped at 0.



With SH the backrest is a deeper green and its floral pattern is clearer; the DC color, a direction average, looks flatter and lighter.

The seat and backrest are again darker with SH, and the white frame is slightly greyer.

Almost no difference: the matte beige underside has small higher-order SH terms.
B.7.2 Training on a Harder Scene
| Materials, validation split | Iterations | PSNR | SSIM |
|---|---|---|---|
| Baseline (Q6 setup, random init) | 1000 | 18.49 | 0.682 |
| Improved | 2000 | 22.95 | 0.848 |




Improved learning rates:
| Parameter | Learning rate |
|---|---|
| Opacities (pre-activation) | 0.05 |
| Scales (pre-activation) | 0.005 |
| Colors | 0.02 |
| Means | 0.002, decayed exponentially to 0.00002 |
| Quaternions | 0.001 |
Modifications compared to Q6:
- Wider init: initial means scaled by 3; the default (std 0.2) is too small for this scene.
- Anisotropic Gaussians: per-axis scales and trained rotations, to fit the curved surfaces.
- L1 + SSIM loss:
0.8 · L1 + 0.2 · (1 - SSIM), 11×11 Gaussian window. - Means learning rate: 4× the Q6 rate, decayed exponentially to 1%.
+4.5 dB PSNR and +0.17 SSIM. The baseline leaves the outer spheres as glowing blobs, since all initial points start near the center; the improved run recovers every sphere. Edges stay soft and reflections are missing, because each Gaussian has a single RGB color.