Assignment 3: Neural Volume Rendering and 3D Gaussian Splatting

16-825 Learning for 3D Vision · Yongyi Xiong (andrewid: yongyix)

All experiments were run on a CPU (Apple M4 Max) in PyTorch, so every hard-coded .cuda() call was replaced by a device variable. No other change was made to the provided training or rendering loops unless stated below.

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

get_pixels_from_image maps pixel indices to NDC with x = 2i / (W − 1) − 1 (same for y), so the corner pixels land exactly on ±1, which matches grid_sample(align_corners=True) used later to read ground-truth colours. get_rays_from_pixels places the NDC points on the plane z = 1, lifts them to world space with camera.unproject_points, uses the camera centre as the ray origin, and normalizes the direction so that distances along a ray are metric.

xy grid
xy grid
ray directions
Ray directions (|d| as RGB)

Both images match the reference outputs. As a sanity check, the centre ray direction is (0, 0, 1) and every corner ray is (±0.577, ±0.577, 0.577), i.e. 1/√3 per component for a 90° field of view.

1.4 Point Sampling

StratifiedRaysampler places n_pts_per_ray depths uniformly in [min_depth, max_depth] and computes p = o + t·d for every ray by broadcasting. The figure shows all 65,536 × 64 samples of the first camera. The far boundary is an arc, not a line, because the samples are placed at equal distance from the camera along unit directions.

sample points
Sample points along the rays of the first camera.

1.5 Volume Rendering

For each ray with samples ti:

αi = 1 − exp(−σiΔti),   Ti = ∏j<i(1 − αj),   wi = Tiαi
C = ∑ wi ci,   D = ∑ wi ti

T is computed with an exclusive cumulative product (a column of ones is prepended to 1 − α before torch.cumprod), and _aggregate is a weighted sum over the sample dimension, which is reused for both colour and depth.

part 1 gif
Spiral rendering of the box.
depth
Depth, normalized by its maximum.

2. Optimizing a Basic Implicit Volume

get_random_pixels_from_image draws a random permutation of the pixel indices and keeps the first n_pixels (sampling without replacement). The loss is the mean squared error between rendered and ground-truth colours. Training uses 1000 epochs over the 4 views with 4096 rays per step.

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
part 2 gif
Spiral rendering of the optimized box.
comparison with reference
Top: our spiral rendering at frames 0, 5, 10, 15. Bottom: the reference GIF at the same frames.

Comparison with the reference GIF. The two GIFs are nearly identical. Across the 20 frames, the foreground area of our box (pixels whose R + G + B exceeds 30 on the 0 to 255 scale) is 98.5% to 100.2% of the reference (mean 99.2%), the bounding boxes differ by at most one pixel, and the mean absolute pixel difference is 0.3%, which is at the level of GIF colour quantization.

before after gt
Training views. Top: before training. Middle: after training. Bottom: ground truth.

3. Optimizing a Neural Radiance Field

Architecture. The 3D position is encoded with HarmonicEmbedding (6 frequencies, 39 dimensions with the raw input). A 6-layer MLP with 128 hidden units and a skip connection that re-injects the embedding at layer 3 (MLPWithInputSkips) produces a 128-d feature, followed by a ReLU. Density is ReLU(Linear(feature)) and colour is Sigmoid(Linear(feature)). The loss is MSE; training follows nerf_lego.yaml (250 epochs, 128×128, 1024 rays and 128 samples per ray, lr 5e-4 with step decay).

part 3 gif
Spiral rendering of the lego NeRF after 250 epochs.
comparison with reference
Top: our frames 0, 5, 10, 15. Bottom: the reference GIF at the same frames (mean PSNR 31.8 dB between the two GIFs).

Initialization detail: with a ReLU on the density output, roughly one third of random initializations produce zero density for every sample, so the gradient is exactly zero and the model never trains. I later initialized the density head bias to 0.1, which removed this failure in 200 out of 200 seeds. The lego result above was trained before this change and did not hit the degenerate case.

4.1 View Dependence

Method. Density depends only on position, as in Q3. For colour, the 128-d position feature is concatenated with the harmonic embedding of the viewing direction (2 frequencies, 15 dimensions) and passed through a small head Linear(143, 64) → ReLU → Linear(64, 3) → Sigmoid. The model is trained on the materials scene with the settings of nerf_materials.yaml (128×128, 250 epochs, lr 1e-4).

part 4.1 gif
View-dependent NeRF on the materials scene. The specular highlights on the metallic spheres move with the viewpoint.

Trade-offs. View dependence lets the model reproduce effects such as the highlights on the metal spheres, which a position-only colour has to average out. The cost is that the network gains an easy way to fit the training images without learning consistent geometry: it can paint each training view with a different colour instead of placing surfaces correctly, which looks good on training views and breaks on novel ones. Three choices in the design limit this. Density never sees the direction, so geometry is shared by all views (I checked that flipping the direction leaves density unchanged and only changes colour). The direction uses a low-frequency embedding (2 frequencies), so colour can only vary smoothly with the viewpoint. The direction branch is small (one 64-unit layer), so most of the capacity stays in the view-independent trunk. More view dependence (more frequencies or a larger branch) would capture sharper reflections but increases the risk of overfitting, especially with few training views.

4.2 Coarse/Fine Sampling

Method. Two NeRF networks with the Q3 architecture are trained jointly. The coarse pass uses 64 uniformly spaced samples per ray and renders with the coarse network. Its weights (excluding the first and last sample, plus 1e-5) are normalized into a piecewise-constant PDF over the bins between sample midpoints, and 128 additional depths are drawn by inverse transform sampling (torch.searchsorted on the CDF with linear interpolation inside the bin). The fine network then renders all 192 sorted samples. The loss is the sum of the coarse and fine MSE, and the fine samples are detached so the coarse network is only supervised by its own rendering.

part 4.2 gif
Coarse/fine NeRF on lego (fine output).
Q3 vs 4.2
Top: Q3 (single network). Bottom: coarse/fine. Same frames.
Network evaluations per rayTime / epochTotal timePSNR vs reference GIF
Q3 (uniform, 128 samples)128~15 s75 min31.80 dB
4.2 (64 coarse + 192 fine)256~23 s110 min31.38 dB

Trade-offs. Hierarchical sampling spends samples where the coarse pass found mass, so in principle it resolves surfaces more finely than uniform sampling with the same number of fine samples. It roughly doubles the work per ray, since two networks are evaluated and the fine network sees 192 points, and it made training about 1.5× slower here. At this low resolution (128×128) the visual difference to Q3 is small. Two caveats on the comparison: the reference GIF is itself a NeRF rendering rather than a ground-truth photo, so the PSNR column measures agreement with the reference implementation, and the two models do not use the same total number of samples. The benefit of coarse/fine sampling should grow at higher resolution or when the sample budget per ray is small, which is when uniform samples miss thin structures.

4.3 VolSDF

Implementation

Results and the settings that made them work

All three versions use α = 10, β = 0.05, 128 samples per ray and 250 epochs. They differ in how the network is initialized and how colour is predicted.

VolSDF versions
Columns: (1) no geometric init, (2) geometric init, (3) geometric init + geometry features, (4) reference. Rows 1 and 2: geometry at two viewpoints. Rows 3 and 4: colour.
VersionColour PSNR vs reference GIFGeometry silhouette IoU vs reference
(1) Independent colour MLP, default init30.24 dB0.24
(2) + geometric (sphere) initialization23.58 dB0.85
(3) + colour from geometry features33.24 dB0.77

(1) Default initialization. In this run the initial network output was a distance of about −0.01 everywhere (the sign depends on the random seed), so the whole volume started "inside" with a density of 10·(1 − ½e−0.01/0.05) ≈ 5.9, more than half of α. Because the background of every training image is black, a black opaque wall explains the background just as well as empty space, and training never carves out the regions behind and below the object. The colour looks fine, but the extracted geometry is a large slab with the bulldozer embedded in it.

(2) Geometric initialization. The distance MLP is initialized so that it outputs approximately ‖x‖ − 1 (the IGR/SAL scheme: hidden weights N(0, 2/out), output weights with mean √(π/in) and bias −r, and the sine/cosine inputs zeroed in the first layer). Space starts empty outside a unit sphere and training only has to sculpt the sphere into the object. The geometry becomes clean, but the colour MLP, which is shallow (one hidden layer) and reads the high-frequency position encoding directly, paints coloured stripes in the thin shell outside the surface where density is small but non-zero.

(3) Geometry features. The distance MLP outputs 1 + 128 values: the first is the signed distance and the remaining 128 form a feature vector that is fed, together with the position encoding, to the colour MLP (the IDR/VolSDF design). Geometric initialization is applied only to the distance row. Colour now follows the surface, the stripes disappear, and the colour PSNR rises by almost 10 dB. Because the colour loss now also updates the shared trunk, the geometry is slightly worse than in (2) around the base plate. Version 3 is my best model overall because of its much better colour, while version 2 has the cleanest geometry.

4.3 geometry
Geometry of the best overall model (version 3).
4.3 colour
Colour rendering of the best overall model (version 3).
4.3 geoinit geometry
Version 2 geometry.
4.3 geoinit colour
Version 2 colour.
4.3 no init geometry
Version 1 geometry.

Hyperparameter choices

SettingValueReason
α10With 128 samples in [2, 6], samples are 0.031 apart. Inside the object each sample then has opacity 1 − exp(−10 × 0.031) ≈ 0.27, so a ray becomes nearly opaque within about ten samples (0.3 units) after crossing the surface. This is solid enough for the surface to occlude what is behind it, while a single sample does not saturate.
β0.05The density goes from 0.9α to 0.1α over about 3.2β = 0.16 units, which spans about five samples. A much smaller β would make the transition thinner than the sample spacing, so samples could step over the surface and receive little gradient.
Eikonal weight0.02Keeps ‖∇d‖ close to 1 as a regularizer without dominating the colour loss.
Geometric init radius1.0Comparable to the size of the object. It only has to start with empty space outside; training reshapes the sphere into the bulldozer.
Geometry feature size128Equal to the hidden width of the distance MLP.

α, β and the eikonal weight are the values provided in volsdf_surface.yaml; I did not sweep them. My experiments instead varied the initialization and the colour branch, which turned out to be the factors limiting the results above.

What α and β do

Ψβ(−d) goes from 0 far outside the surface to 1 deep inside it and equals 1/2 on the surface. α scales this, so it is the density of the interior: deep inside the object σ ≈ α, on the surface σ = α/2, and a larger α makes the solid more opaque per unit length. β is the width of the transition: the density falls from α to 0 over a band of a few β around the zero level set. As β → 0 the density becomes a step function at the surface, and a large β turns the surface into a thick fog.

  1. How does high β bias the learned SDF? What about low β? With a high β the density is smeared over a wide band, so a ray starts accumulating opacity well before the zero level set and the rendering weights spread out along the ray. The network can then match the images with a blurry, thick transition, and the zero level set is only loosely tied to the true surface; fine and thin parts tend to be smoothed away or shifted. With a low β the density is close to a step and the weights concentrate at the zero crossing. A lower β encourages a sharper transition and tighter alignment with the surface, but does not guarantee accurate geometry, and regions far from the zero level set receive little supervision.
  2. Would an SDF be easier to train with low or high β? High β. The density, and hence the rendered colour, changes smoothly with the predicted distance over a wide band, so many samples along each ray produce useful gradients even when the initial surface is far from the true one. With a low β the density is essentially constant except in a thin band around the zero level set, so most samples get almost no gradient and the optimization can stall, the same reason the box in Q2 needed a soft boundary.
  3. Would you be more likely to learn an accurate surface with high or low β? Low β, provided training is stable. When the density switches sharply at the zero level set, the image loss constrains the position of that level set more tightly; a high β leaves the surface position more ambiguous within the transition band. In practice a schedule that starts with a larger β and decreases it (or learning β, as VolSDF does) combines easy early training with an accurate final surface.
B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

All four unit tests pass, and frame 29 of the GIF matches the provided reference frame (PSNR 74.9 dB on the RGB panel).

part 5 gif
Rendering of the pre-trained chair Gaussians (RGB, depth, silhouette).

5.x A Note on the World-to-Camera Rotation W

The submitted results use W = camera.R, as instructed. While implementing 5.1 I found that the geometrically correct choice is W = camera.RT.

Derivation. PyTorch3D uses row vectors: x_cam = x_world R + T. The covariance formula Σ' = WΣWT uses column vectors, x_cam = W x_world. Transposing the PyTorch3D form gives x_camT = RT x_worldT, so W = RT.

Monte Carlo check. For the five anisotropic Gaussians of unit test 2 (scaled down so the linearization holds), I sampled 200,000 points from each 3D Gaussian, projected them with camera.transform_points_screen, and measured the covariance of the projected pixels. With W = RT the predicted 2D covariance matches the empirical one within 0.8%; with W = R the error reaches 131% and the sign of the off-diagonal term is flipped for several Gaussians, i.e. the ellipses are tilted the wrong way.

Effect on renders. The pre-trained chair was optimized with the correct projection in the original codebase, so W = RT reproduces it more faithfully: with W = R, flat Gaussians are rotated off the surface and appear as thin streaks.

W comparison frame 29
Frame 29, cropped. Left: W = R (submitted). Middle: W = RT. Right: provided reference. W = R matches the reference exactly, including the white streaks along the seat edge; W = RT gives clean edges.
W comparison frame 22
Frame 22. Left: W = R. Right: W = RT. The streaks around the base and legs disappear with the correct rotation.

With W = RT, unit tests 2 and 4 fail because their expected values were generated with W = R. Isotropic Gaussians (Q6 and Q7.2) are unaffected: Σ = s²I and W(s²I)WT = s²I for any rotation, which I also confirmed numerically (maximum difference 1.2e-7).

6. Training 3D Gaussian Representations

Means, scales, colours and opacities are made trainable; quaternions are not, since the Gaussians are isotropic and a rotation has no effect. The loss is L1 between the rendered and ground-truth images (black background).

ParameterLearning rate
Opacities (pre-activation)0.01
Scales (log)0.001
Colours0.01
Means0.0001

The means get the smallest learning rate because they are initialized from the provided point cloud and are already close to the right positions.

IterationsMean PSNR (test)Mean SSIM (test)
100028.200.930
training progress
Training progress. Top: renders of the Gaussians. Bottom: ground truth.
final renders
Final renders of the training views.

7.1 Rendering Using Spherical Harmonics

colours_from_spherical_harmonics evaluates all 16 real SH basis functions up to degree 3 for every Gaussian in a vectorized way and returns clamp(SH·Y(d) + 0.5, 0, 1), where d is the normalized direction from the camera centre to the Gaussian mean. It matches the eval_sh function of the original 3DGS code to 2e-7 on random inputs, and keeping only the degree-0 coefficients reproduces the DC colours used in Q5 exactly. A --use_sh flag in render.py switches between the two colour models.

Q5 gif
Q5: DC colour only.
Q7.1 gif
Q7.1: full spherical harmonics.

Side-by-side comparisons (left: DC only, right: full SH)

frame 14
Frame 14 (largest difference, mean absolute difference 0.031).

The fabric on the back and seat is darker and the golden pattern stands out much more with SH. The DC colour is the average over all viewing directions, which washes out the sheen of the fabric and makes it look flat and bright; the higher-order terms restore how the fabric reflects light towards this particular viewpoint.

frame 3
Frame 3, the viewpoint of the reference SH image (mean absolute difference 0.022).

Shading on the back and seat varies more with SH, and the carved golden trim on the frame looks glossier. These are view-dependent effects, so the DC-only render cannot express them.

frame 22
Frame 22, seen from below (mean absolute difference 0.008).

Almost no difference. This view mostly shows the matte wooden underside, which is close to diffuse, so its colour hardly depends on the viewing direction and the higher-order coefficients are small. (The streaks around the legs come from W = R, see Section 5.x.)

7.2 Training on a Harder Scene

Baseline

The baseline uses the Q6 setup unchanged: 10,000 isotropic Gaussians with init_type="random", the Q6 learning rates, L1 loss and 1000 iterations. All runs in this section share the same random seed, the same order of training views, and are evaluated on the same 100 validation views.

baseline progress
Baseline training progress. Top: renders. Bottom: ground truth.

Best method

best progress
Best method (A+L) training progress.
baseline final
Baseline final renders.
best final
A+L final renders.
Method (1000 iterations)PSNRSSIM
Baseline (Q6 setup)16.270.616
A+L (best)21.230.672

In the baseline the Gaussians never leave the small cluster around the origin where they are initialized, so every sphere is missing. With A+L every sphere appears at the correct position and both metrics improve; the remaining errors are blur and missing material detail, which is expected from 10,000 isotropic Gaussians and 1000 iterations.

Modifications in the best method

  1. Initialization (A).
    • Means are drawn uniformly inside a ball of radius 1.4 (radius r·u1/3 along a random direction) instead of N(0, 0.2²). The radius was chosen from the coverage check below.
    • Opacity starts at 0.1 instead of sigmoid(8) ≈ 0.9997, so the random colours in front do not completely block the Gaussians behind them.
    • Each scale is the RMS distance to the 3 nearest neighbours (knn_points with K = 4, excluding the point itself), as in the original 3DGS, so the Gaussians tile the volume without large gaps.
  2. Position learning rate (L). Means use lr 0.001 instead of 0.0001, since they start far from their targets. All other learning rates, the loss and the number of Gaussians are unchanged.

Choosing the initialization radius

I initialized the Gaussians with each radius, rendered 20 training views, and measured the fraction of ground-truth object pixels covered by the initial silhouette (mask > 0.3).

RadiusObject pixels covered (mean)Worst viewCovered area that is background
1.089.7%84.4%61%
1.3100.0%99.9%72%
1.5100.0%100.0%74%
2.0100.0%100.0%75%

A radius of 1.0 misses the outer spheres. Larger radii spread the same 10,000 Gaussians over more empty space, so I used the smallest radius that covers the scene plus a small margin (1.4).

initialization
Before training. Top: baseline initialization. Middle: initialization A. Bottom: ground truth.

Round 1: ablation of A and L (300 iterations)

RunInitMeans lrPSNRSSIM
Bbaseline1e-416.120.607
Anew1e-418.590.462
Lbaseline1e-316.250.614
A+Lnew1e-319.080.504
round 1
After 300 iterations. Rows: B, A, L, A+L, ground truth.

L alone changes nothing visible: a 10× larger step is still too small for points that start in a tight cluster. Initialization is what matters, and the larger learning rate helps only once the Gaussians start in the right region (A+L > A). The SSIM of B and L is misleading here: about 75% of each image is black background, and a render that is almost entirely black scores well on those regions even though every object is missing. At 300 iterations A and A+L were still clearly improving while B had plateaued, so A+L was continued to 1000 iterations.

Round 2: additional changes on top of A+L (1000 iterations)

RunPSNRSSIM
Baseline16.270.616
A+L21.230.672
A+L + exponential decay of the means lr (1e-3 to 1e-4)21.130.666
A+L + decay + SSIM loss (0.8·L1 + 0.2·(1 − SSIM))21.050.621
round 2
After 1000 iterations. Rows: baseline, A+L, A+L + decay, A+L + decay + SSIM, ground truth.

Neither learning-rate decay nor the SSIM loss improved the result within 1000 iterations. With the decay, the PSNR difference was approximately 0.1 dB; more seeds are needed to assess its significance. The SSIM loss lowered validation SSIM by 0.045. I did not run experiments to isolate the causes, so the following are possible explanations rather than conclusions:

A repeat of the baseline without a fixed seed gave 16.26 / 0.617, differing by 0.01 dB from the seeded run. This single repeat suggests that seed variation is small compared with the 5 dB gap between the baseline and A+L, although more seeds would be needed to quantify it.