Assignment 3 for 16-825 (Volume Rendering, NeRF, and 3D Gaussian Splatting)

Kartik Sharma
Andrew ID: kartiksh

Experimental setup

A. Neural Volume Rendering

1.3 Ray sampling (5 points)

get_pixels_from_image maps the pixel indices to the range [-1, 1] in x and y. get_rays_from_pixels appends z = 1 to each pixel coordinate, lifts the points to world space with camera.unproject_points, and sets each ray’s direction to the normalized vector from the camera centre to its lifted point.

Pixel grid visualization
xy grid (vis_grid)
Ray direction visualization
Ray directions (vis_rays)

1.4 Point sampling (5 points)

StratifiedRaysampler places n_pts_per_ray depths between min_depth and max_depth. The sample points are origin + depth × direction.

Sample points along the rays of the first camera
Sample points along the rays of the first camera, rendered with render_points.

1.5 Volume rendering (20 points)

The weight of sample i is Ti(1 − exp(−σiΔti)), where the transmittance Ti = ∏j<i exp(−σjΔtj) and T1 = 1. Color and depth are the weighted sums of the per-sample color and the per-sample depth (sample_lengths).

Spiral rendering of the box
part_1.gif
Depth map of the box
Depth, normalized to [0, 1]

2. Optimizing a basic implicit volume (10 points)

2.1 Random ray sampling. get_random_pixels_from_image draws n_pixels pixels without replacement from the full grid with torch.randperm, on the GPU. 2.2 Loss. Mean squared error between the rendered colors and the ground-truth colors rgb_gt.

After training with train_box:

xyz
Box center0.250.250.00
Box side lengths2.011.501.51
Spiral rendering of the optimized box
Mine: part_2.gif
TA spiral rendering of the optimized box
TA reference

3. Optimizing a Neural Radiance Field (20 points)

Network. The 3D position goes through a harmonic embedding (6 frequencies) and then an MLP with 6 layers of 128 units and ReLU activations. The embedded input is concatenated again before layer 3 (a skip connection). The last layer outputs 4 values: a ReLU maps the first to density and a sigmoid maps the other three to RGB. Loss. MSE between the rendered and ground-truth colors. Training. The nerf_lego defaults: 250 epochs, 1024 rays per batch, 128 samples per ray, learning rate 5e-4 decayed ×0.8 every 50 epochs.

Spiral rendering of the lego NeRF
part_3.gif. PSNR 24.29 on the training views, 23.88 on the test views.

4.1 View dependence (10 bonus points)

Network. As in the NeRF paper, density depends on position only. The 6-layer position MLP outputs a 128-dimensional feature; after a ReLU, a linear layer maps it to density. The feature is concatenated with the harmonic embedding of the normalized ray direction (2 frequencies) and passed through a small color head (one hidden layer of 64 units), which outputs RGB. Training. The nerf_materials settings (250 epochs, learning rate 1e-4) with view_dependence=True. The baseline is the same model with view_dependence=False.

Materials scene with view dependence
With view dependence
Materials scene without view dependence
Without view dependence
Materials sceneTrain PSNRTest PSNR
Without view dependence23.2622.75
With view dependence23.9923.46

Trade-offs. View dependence lets the color change with viewing direction, so the model can represent the specular highlights on the shiny spheres; a model without it can only average them into a dull color. Both train and test PSNR rise by about 0.7 dB, and the gap between them stays the same (about 0.5 dB), so this model does not overfit. The risk grows with how much the network may use the direction. If the direction also fed the density branch, or used many frequencies, the network could explain each training image with view-specific color or geometry instead of one consistent 3D scene, which produces floaters and artifacts at novel views. Keeping density a function of position only, using a low-frequency direction embedding and a small color head limits view dependence to smooth effects such as highlights.

4.2 Coarse/fine sampling (10 bonus points)

Method. Two NeRFs with the architecture from part 3. The coarse network is evaluated at 64 stratified samples per ray. Its rendering weights, normalized, form a piecewise-constant probability distribution along the ray, and inverse-CDF sampling draws 128 extra depths from it. The fine network is evaluated at all 192 depths in sorted order, and its output is the final rendering. Loss. MSE of the coarse rendering + MSE of the fine rendering; the coarse loss keeps the coarse weights a useful sampling distribution. The fine samples are drawn from detached coarse weights. The config is nerf_lego_hierarchical.yaml; the other settings match part 3.

Spiral rendering of the coarse/fine lego NeRF
part_4_2.gif (fine network)
LegoSamples per rayTrain PSNRTest PSNRTime per iteration
NeRF (part 3)12824.2923.8819 ms
Coarse/fine64 + 12824.5624.1929 ms

Trade-offs. Coarse/fine sampling gains 0.3 dB of test PSNR. It puts most samples near the surfaces that contribute to the pixel instead of in empty space, which sharpens thin structures such as the bulldozer’s tracks and arm. The cost is a second network (twice the parameters) and about 1.5× the time per training iteration (forward + backward, 1024 rays, measured on one H200). At a fixed compute budget, uniform sampling with more points is a simpler alternative, but its cost grows with the number of samples everywhere on the ray, while importance sampling spends samples only where the coarse network finds content.

4.3 VolSDF (10 bonus points)

Network. A harmonic embedding of the position (6 frequencies) feeds a 6-layer, 128-unit distance MLP that outputs the signed distance (no output activation, since distances can be negative) and a 128-dimensional feature. A 2-layer color MLP maps the feature, after a ReLU, to RGB through a sigmoid. Eikonal loss. The mean of (‖∇d‖ − 1)² at random points in the bounding box; a true signed distance function has unit-norm gradient everywhere. SDF to density. σ(x) = α Ψβ(−d(x)), where Ψβ is the CDF of a zero-mean Laplace distribution with scale β. I write it with exp(−|d|/β), so that neither branch of torch.where can overflow.

Intuition for α and β. α is the density inside the object: far inside the surface σ → α, and on the surface σ = α/2. β controls how sharply the density falls from α to 0 across the surface. The transition has a width of a few β, so a small β gives a nearly solid surface and a large β a soft, foggy one.

  1. How does high β bias the learned SDF? What about low β? With a high β, density spreads far from the zero level set, so points well outside the object still absorb light. The renderings are blurry, and the learned surface is smoothed, with thin parts thickened or lost. A low β concentrates density at the zero level set, so the model renders a sharp surface close to the true one, but only points very near the surface receive gradient.
  2. Would an SDF be easier to train with volume rendering and low β or high β? Why? High β. The density, and therefore the gradient, reaches points far from the current surface, so a randomly initialized SDF receives a training signal everywhere along each ray and the surface can move a long way. With a low β, the weights are nearly a delta at the surface, most samples get almost no gradient, and optimization is unstable or stuck early on.
  3. Would you be more likely to learn an accurate surface with high β or low β? Why? Low β. The renderer then matches opaque surface rendering, and the error-bound argument of VolSDF shows the approximation error shrinks with β. With a high β the photometric loss can be satisfied by a fuzzy shell around the object, so the zero level set does not have to sit on the true surface. In practice one trains with a moderate β first (or learns β) and lowers it.

Experiments. volsdf_surface settings (α = 10, β = 0.05, 250 epochs, learning rate 5e-4), with two eikonal weights:

VolSDF geometry, eikonal weight 0.02
Geometry, eikonal weight 0.02
VolSDF color, eikonal weight 0.02
Color, eikonal weight 0.02
VolSDF geometry, eikonal weight 0.1
Geometry, eikonal weight 0.1 (best)
VolSDF color, eikonal weight 0.1
Color, eikonal weight 0.1 (best)

Chosen settings. I keep α = 10 and β = 0.05: β = 0.05 is small compared with the scene (about 1/40 of the bulldozer’s size), so the surface is sharp, yet wide enough that training converges from a random initialization. With eikonal weight 0.02, the color renderings are good, but the extracted mesh has holes and speckles: the field is not a valid SDF away from the surface, so the marching-cubes level set breaks up. Raising the eikonal weight to 0.1 forces a unit-norm gradient everywhere, and the bulldozer becomes a solid, watertight mesh. The remaining clutter around the base comes from the black background, which gives no signal that the space below the bulldozer is empty.

B. 3D Gaussian Splatting

Implementation notes. Each 3D covariance is Σ = R S Sᵀ Rᵀ, built from the quaternion and the scales. It is projected to the image as Σ’ = J W Σ Wᵀ Jᵀ, with W = Rᵀ of the PyTorch3D camera, since PyTorch3D multiplies row vectors (xcam = x R + T). The provided unit test expects W = R, so 2 of its 4 cases fail with my code. I kept Rᵀ because the covariance it gives matches a Monte Carlo estimate (Gaussian samples pushed through the camera), and W = R renders visibly streaky splats. The 2D Gaussian is evaluated as an elementwise quadratic form instead of batched 2×2 matrix products, which is 15× faster with the same output.

5. 3D Gaussian rasterization (35 points)

Rendering of the pre-trained chair Gaussians
render.py output (DC colour only)

6. Training 3D Gaussian representations (15 points)

Settings. Isotropic Gaussians initialized from the provided points, L1 loss, Adam, 1000 iterations.

ParameterLearning rate
Opacities0.01
Scales0.005
Colours0.01
Means0.0005
Truck (held-out views)PSNRSSIM
Mine29.560.940
Training progress on the truck scene
Training progress (top: prediction, bottom: ground truth)
Final renders of the trained truck
Final renders

7.1 Rendering using spherical harmonics (10 points)

Method. The colour of each Gaussian is evaluated from its degree-3 spherical harmonics (16 coefficients per channel) in the direction from the camera centre to the Gaussian mean, plus 0.5, clamped at 0. With only the DC coefficient this reproduces the colours from question 5 exactly.

Rendering with DC colour only
Q5: DC only
Rendering with full spherical harmonics
Q7.1: full spherical harmonics

Side-by-side comparisons (left: DC only, right: full SH; same frame):

Frame 4, DC only vs. spherical harmonics
Frame 4

Frame 4: with SH, the backrest and seat fabric show directional shading: the brocade pattern has brighter highlights and darker folds, while the DC version is flat and uniformly lit. The white frame also gets shading on the side facing away from the view.

Frame 12, DC only vs. spherical harmonics
Frame 12

Frame 12: the difference is clearest on the seat. Seen from this angle, the SH seat is darker green with glossy gold specular highlights on the pattern, as fabric with a sheen looks from a grazing angle. The DC render keeps the average colour from all views, so it looks lighter and washed out.

Frame 20, DC only vs. spherical harmonics
Frame 20

Frame 20: from underneath, the two renders are close, since the bottom of the seat is a matte, mostly uniform surface. SH adds slightly stronger contrast on the green side panels and darker shading on the carved wooden parts.

7.2 Training on a harder scene (10 points)

Baseline. The question 6 setup on the materials scene with init_type="random": isotropic Gaussians, the same learning rates, L1 loss, 1000 iterations.

Improved method. Changes relative to question 6:

  1. Anisotropic Gaussians, with quaternions trained at learning rate 0.001, so a Gaussian can stretch along the spheres’ surfaces.
  2. Wider random initialization. The means are scaled 5× at initialization, because the materials spheres spread much wider than the default random point cloud, which is concentrated near the origin.
  3. Loss = 0.8 · L1 + 0.2 · (1 − SSIM), as in the original paper.
  4. Learning rates. Opacities 0.05, scales 0.005, colours 0.01. The means’ learning rate decays exponentially from 1.6e-3 to 1.6e-5 over training: large steps move the random points into place early, small steps refine them later.
  5. Simplified adaptive density control. Every 250 iterations from iteration 500 to 3000, Gaussians whose 3D mean has an average gradient norm above 0.0002 are cloned (if small) or split into two smaller ones (if their largest scale exceeds 0.05), and Gaussians with opacity below 0.005 are pruned. The optimizer is rebuilt after each step. This grows the scene from 10,000 to about 49,000 Gaussians.
  6. Longer training: 6000 iterations.
Materials (held-out views)IterationsPSNRSSIM
Baseline (Q6 setup)100017.990.666
Improved600024.470.897
Baseline training progress
Baseline: training progress
Improved training progress
Improved: training progress
Baseline final renders
Baseline: final renders
Improved final renders
Improved: final renders

The baseline stays a set of blurry blobs: its isotropic Gaussians start bunched near the origin and the fixed learning rates cannot spread them over all the spheres in 1000 iterations. With the improvements, the individual spheres and their specular highlights are resolved, at +6.5 dB PSNR. Density control contributes most of the detail, since it adds Gaussians exactly where the reconstruction error is largest.