Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

16-825 Learning for 3D Vision

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray sampling

get_pixels_from_image places one sample at each pixel centre and maps it to NDC in [−1,1][-1, 1]. get_rays_from_pixels lifts each NDC point to depth 1 and unprojects it to world space with camera.unproject_points(..., from_ndc=True). Each ray starts at the camera centre and points through its unprojected pixel. Directions are normalised, so sample depths are distances along the ray.

xy grid ray directions

1.4 Point sampling

StratifiedRaysampler places n_pts_per_ray evenly spaced depths t∈[tnear,tfar]t \in [t_\text{near}, t_\text{far}] along every ray and returns the points 𝐨+t𝐝\mathbf{o} + t\,\mathbf{d}. The figure shows every tenth sample from the first camera.

Sample points from the first camera

1.5 Volume rendering

Weights follow the emission–absorption model, with T1=1T_1 = 1:

wi=Ti(1−e−σiΔti),Ti=∏j<ie−σjΔtj,C=∑iwiLe,i,D=∑iwiti. w_i = T_i\,\bigl(1 - e^{-\sigma_i \Delta t_i}\bigr), \qquad T_i = \prod_{j<i} e^{-\sigma_j \Delta t_j}, \qquad C = \sum_i w_i\, L_{e,i}, \qquad D = \sum_i w_i\, t_i .

The transmittance is an exclusive cumulative product of the per-segment survival probabilities. The same weights aggregate colour (_aggregate) and depth, so the depth map comes from the same pass. The depth map below is normalised by its maximum value.

Spiral render (part_1.gif) Depth (normalised)

2. Optimizing a basic implicit volume

2.1 Random ray sampling and 2.2 loss

get_random_pixels_from_image takes a random permutation of the full pixel grid and keeps the first n_pixels entries. The loss is the mean squared error between rendered and ground-truth colours. After 1000 epochs (batch 4096, lr 5e-4):

x y z
Box center 0.25 0.25 0.00
Box side lengths 2.01 1.50 1.50

2.3 Visualization

View 0 View 1 View 2 View 3
Before
After
Optimized box (part_2.gif)

3. Optimizing a Neural Radiance Field (NeRF)

Architecture. The input point is positionally encoded with 6 frequencies (39 dims) and passed through a 6-layer, 128-unit ReLU MLP (MLPWithInputSkips). The encoding is concatenated back in at layer 3. Two heads read the trunk features:

View dependence is implemented behind a use_view_dir config flag, but it is off here because this section asks for a position-only model.

Training. The nerf_lego config: 250 epochs at 128×128, 1024 rays per step, 128 samples per ray in [2,6][2, 6], Adam at lr 5e-4 decaying ×0.8 every 50 epochs, and an MSE loss. The final training MSE is about 0.0025.

NeRF on the lego bulldozer (part_3.gif)

The render recovers the bulldozer’s overall shape, the arm and bucket, the tracks, and the texture of the base plate. Since the model has no view dependence, the colours look matte from every angle.

4. NeRF Extras: 4.3 VolSDF

Implementation.

σ={α⋅12e−d/βd>0α⋅(1−12ed/β)d≤0 \sigma = \begin{cases} \alpha \cdot \tfrac12 e^{-d/\beta} & d > 0 \\[2pt] \alpha \cdot \bigl(1 - \tfrac12 e^{d/\beta}\bigr) & d \le 0 \end{cases}

The code evaluates this as 0.5 * exp(-|d| / β) so that the exponential never overflows.

Geometry (part_4_3_geometry.gif) Colour (part_4_3.gif)

Settings. I used the provided volsdf_surface settings: α = 10, β = 0.05, eikonal weight 0.02, 250 epochs, and lr 5e-4. With β = 0.05 the surface transition is about 1–2% of the scene size, sharp enough to resolve the bucket, cab and tracks. With α = 10 the interior becomes opaque within a few of the 128 samples per ray. The remaining artifacts are small floaters near the base plate, where the training views give little constraint.

What α and β do. α is the density deep inside the object: as −d-d grows, Ψβ→1\Psi_\beta \to 1, so interior points get density α. It also caps how opaque a thin shell can become. β is the scale of the Laplace CDF, so it sets the width of the transition from empty (density ≈ 0) to solid (density ≈ α) around the zero level set. Small β gives a sharp, almost binary surface. Large β gives a soft, foggy shell several β thick.

  1. How does high β bias the learned SDF? What about low β? With high β, density leaks well outside the surface and stays low just inside it. The rendered surface therefore sits in front of the true zero level set, and the network can explain the images with blurry, over-smoothed geometry: thin parts and sharp edges get rounded away. With low β, density approaches an indicator function of the interior. The renderer then sees only the zero crossing, which pushes the zero level set onto the true surface, but only points very close to the surface receive gradient.

  2. Would an SDF be easier to train with low β or high β? Why? High β is easier. Every sample within a few β of the surface gets non-zero weight and gradient, so even a poorly initialised SDF gets a learning signal from distant samples, and the loss surface is smooth. With low β the density is close to a step function. Samples that miss the thin transition band receive no gradient, so optimisation stalls easily and becomes sensitive to sample spacing. This is why VolSDF and NeuS anneal β from large to small, or learn it.

  3. Would you be more likely to learn an accurate surface with high β or low β? Why? Low β, as long as training still converges. With a sharp transition, colour is attributed to points right at d=0d = 0, so the zero level set has to lie on the real surface to explain the images. With high β, many surface placements explain the images equally well, and the recovered surface is blurrier.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

5.1 Projection. The 3D covariance is Σ=RSS⊤R⊤\Sigma = R S S^\top R^\top (quaternion_to_matrix for RR). For isotropic Gaussians it is simply s2Is^2 I, and the quaternion is never touched. The 2D covariance is Σ′=JWΣW⊤J⊤\Sigma' = J W \Sigma W^\top J^\top, using the provided Jacobian and the world-to-view rotation WW. Means are projected to pixels with camera.transform_points_screen.

5.2 Evaluation. P=−12(𝐱−𝛍)⊤Σ′−1(𝐱−𝛍)P = -\tfrac12 (\mathbf{x} - \boldsymbol\mu)^\top \Sigma'^{-1} (\mathbf{x} - \boldsymbol\mu), evaluated for all Gaussians and pixels at once with a single einsum.

5.3 Filter and sort. Depth is the view-space zz of each mean. The Gaussians are sorted by argsort, and those with z<0z < 0 are dropped.

5.4 Alphas and transmittance. αi=oiexp⁡(Pi)\alpha_i = o_i \exp(P_i) and Ti=∏j<i(1−αj)T_i = \prod_{j<i}(1-\alpha_j). The transmittance is a cumprod over the batch, with the previous mini-batch’s final transmittance prepended.

5.5 Splatting. With wi=αiTiw_i = \alpha_i T_i: image =∑iciwi= \sum_i c_i w_i, depth =∑iziwi= \sum_i z_i w_i, and silhouette =∑iwi= \sum_i w_i.

Pre-trained chair: colour, depth, silhouette (part_5_render.gif)

6. Training 3D Gaussian Representations

10,000 isotropic Gaussians are initialised from the provided point cloud. Means, log-scales, colours and opacity logits are trainable. Quaternions are not, since rotation has no effect on an isotropic Gaussian. Each step renders one training view on a black background and applies an L1 loss. The optimiser is Adam.

Parameter Learning rate
Opacities (pre-sigmoid) 0.05
Colours 0.02
Scales (log) 0.005
Means 0.0005

The means get the smallest learning rate: the point cloud already places them well, and large steps scatter them. Opacity logits start at 8, where the sigmoid is saturated, so they need a large learning rate to move at all.

Training progress, predictions (top) and ground truth (bottom), every 20 iterations (part_6_training_progress.gif)
Final renders of the training views (part_6_training_final_renders.gif)

7. Extensions: 7.1 Rendering Using Spherical Harmonics

colours_from_spherical_harmonics reshapes the 48 coefficients per Gaussian to 16×316 \times 3 (degree 0–3, RGB). It evaluates the 16 real SH basis functions at the unit direction from the camera centre to each Gaussian (calculate_gaussian_directions), and returns max⁡(0,∑kYk(𝐝)ck+0.5)\max(0,\ \sum_k Y_k(\mathbf{d})\, c_k + 0.5). All Gaussians are processed in one vectorised pass. render.py --use_sh enables it and writes part_7_1_render.gif.

5.5: DC only (part_5_render.gif) 7.1: full SH (part_7_1_render.gif)

Side-by-side comparisons (left: DC only, right: full SH, same frame).

Frame 3

Without SH, the seat and back are a flat mid-green. With SH, the fabric under the backrest and on the inner seat is darker. This is the shading and self-occlusion present in the training images. The damask pattern also has more contrast.

Frame 10

At this grazing angle into the seat, the SH render is noticeably darker and more saturated, and the golden embroidery stands out. The DC colour is an average over all viewing directions, which washes out this view-dependent sheen of the silk.

Frame 20

From below, the two renders differ little because the cream base is matte in both. SH adds slightly warmer highlights on the gilded carving and the armrest scrolls. Diffuse surfaces gain little from higher-order terms, while specular trim gains the most.