1.3 Ray Sampling 5 pts

Implement get_pixels_from_image and get_rays_from_pixels in ray_utils.py, and visualize xy_grid and the rays with vis_grid / vis_rays.

Pixel coordinates are mapped to NDC in [-1, 1] on both axes. Each NDC point is placed on the image plane at depth 1, unprojected to world space, and the ray direction is the normalized vector from the camera center to that point:

x, y = torch.arange(W), torch.arange(H)
x, y = x / (W - 1) * 2 - 1, y / (H - 1) * 2 - 1

ndc_points   = cat([xy_grid, ones], -1)        # z = 1
world_points = camera.unproject_points(ndc_points, world_coordinates=True, from_ndc=True)
rays_o = camera.get_camera_center().expand(N, -1)
rays_d = F.normalize(world_points - rays_o, dim=-1)

1.4 Point Sampling 5 pts

Implement StratifiedRaysampler in sampler.py and visualize the sample points from the first camera with render_points.

Depths are spaced uniformly between min_depth and max_depth and broadcast against every ray to get an (N, P, 3) tensor of sample points:

z_vals = torch.linspace(min_depth, max_depth, n_pts_per_ray).view(1, -1, 1) # (1, P, 1)
sample_points = origins[:, None, :] + directions[:, None, :] * z_vals       # (N, P, 3)
sample_lengths = z_vals * ones_like(sample_points[..., :1])                 # (N, P, 1)
Point cloud of ray samples forming the camera's viewing frustum, seen from the side
Figure 3. Sample points along every ray of the first camera.

1.5 Volume Rendering 20 pts

Implement VolumeRenderer._compute_weights and _aggregate, and render a depth map in addition to color. Visualize the spiral rendering and the depth (normalized by its maximum).

Transmittance at sample i is an exclusive running product, which I compute in parallel as a cumulative sum in log space instead of looping over samples:

sd = density * deltas                                      # (N, P, 1)
sd_excl = cat([zeros_like(sd[:, :1]), sd[:, :-1]], dim=1)  # shift right: T_0 = 1
T = exp(-cumsum(sd_excl, dim=1))
weights = T * (1 - exp(-density * deltas))

feature = sum(weights * rays_feature, dim=1)               # (N, P, C) -> (N, C)
depth   = sum(weights * depth_values, dim=1)               # expected depth

2. Optimizing a Basic Implicit Volume 10 pts

(2.1) Implement random ray subsampling in get_random_pixels_from_image. (2.2) Use an MSE loss to optimize the box's center and side lengths, and report them rounded to two decimals. (2.3) Attach the spiral rendering of the optimized volume.

# 2.1: random subset of pixels per iteration
idx = torch.randperm(xy_grid.shape[0])
xy_grid_sub = xy_grid[idx][:n_pixels]

# 2.2: MSE between rendered and ground-truth colors
loss = criterion(rgb_gt, out['feature'])
xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Spiral rendering of the optimized rainbow box
Figure 6. Spiral rendering of the optimized box.

3. Optimizing a Neural Radiance Field 20 pts

Implement the NeRF MLP in NeuralRadianceField (density via ReLU, color via Sigmoid, optional positional encoding), fill in the loss in train_nerf, and train on the lego bulldozer (--config-name=nerf_lego).

Sample points are positionally encoded (6 frequencies, 39 dimensions) and passed through the provided MLPWithInputSkips trunk: 6 layers, 128 hidden units, with the encoding concatenated back in before layer 3 as in the NeRF paper. Two small heads read the trunk features:

h = MLPWithInputSkips(n_layers=6, input_dim=39, 
                      hidden_dim=128, input_skips=[3])(emb, emb)
density = ReLU(Linear(128, 1)(ReLU(h)))                               # density >= 0
color   = Sigmoid(Linear(64, 3)(ReLU(Linear(128, 64)(ReLU(h)))))      # color in [0, 1]

loss = mse_loss(out['feature'], rgb_gt)

Trained for 250 epochs on 128×128 images, 128 samples per ray. Test-set PSNR: 24.51 dB.

Spiral rendering of the trained lego bulldozer NeRF
Figure 7. Spiral rendering of the trained NeRF.

4.1 View Dependence 10 bonus

Make emission depend on viewing direction, show results on the materials scene, and discuss the trade-off between view dependence and generalization.

The ray direction is encoded (2 frequencies) and concatenated with the trunk features only at the input of the color head; density still depends on position alone. Letting density see the direction would let the network explain each training view with different geometry.

dir_emb = harmonic_embedding_dir(normalize(rays_d))          # (N, D)
dir_emb = dir_emb[:, None].expand(N, P, -1).reshape(N * P, -1)
density = density_head(h)                                    # position only
color   = color_head(cat([ReLU(h), dir_emb], -1))            # position + direction

Training stabilizer. On the materials scene, my first view-dependent runs collapsed. Every density pre-activation went negative (max −0.02), so the ReLU gave zero density and zero gradient everywhere, and the model rendered all black (test PSNR 14.3 dB). The scene is mostly black background, so "no density anywhere" is an easy early local minimum. I kept the required ReLU and added NeRF-style Gaussian noise to the raw density during training only (density_noise_std = 1.0, the config field that already existed), and used the same setting for every row below so they are comparable.

Model (materials, 250 epochs)Train PSNRTest PSNR(Train - Test) gap
View-independent (Q3 model)23.4323.160.27
View-dependent, 2 direction frequencies24.2523.570.68
View-dependent, 6 direction frequencies24.3123.530.78
View-dependent, 2 frequencies, 400×40024.5123.710.80

Trade-off. Adding view dependence improves test PSNR by about 0.4 dB, because the spheres' specular highlights can now move with the camera instead of being averaged into a dull base color. But the train-test gap also grows (0.27 → 0.68 dB), and making the direction branch more expressive (6 frequencies instead of 2) raises train PSNR while slightly lowering test PSNR, with the gap growing to 0.78 dB. That is overfitting: with enough directional capacity, the network can "paint" view-specific colors to match each training image instead of getting geometry and material right, and those painted colors don't carry over to new viewpoints. Keeping density position-only and the direction encoding low-frequency is what keeps that in check.

4.2 Coarse/Fine Sampling 10 bonus

Implement NeRF's coarse/fine strategy: a coarse network estimates geometry, which then guides where the fine network samples. Discuss the speed/quality trade-off.

Two independent NeRF MLPs. The coarse network renders 64 stratified samples per ray. Its weights (minus the first and last) define a per-ray PDF over depth; I draw 128 extra depths from it by inverse-transform sampling (torch.searchsorted on the CDF, then linear interpolation inside the bin). The fine network renders the sorted union of all 192 depths. Both outputs are supervised: loss = MSE(fine) + MSE(coarse).

bins = midpoints(z_coarse)                          # (N, P_c - 1)
pdf  = (w_coarse[:, 1:-1] + 1e-5).normalize()       # detached
cdf  = cat([0, cumsum(pdf)])
u    = rand(N, P_f)                                  # linspace at eval time
idx  = searchsorted(cdf, u, right=True)
z_fine = lerp(bins[idx - 1], bins[idx], (u - cdf[idx - 1]) / (cdf[idx] - cdf[idx - 1]))
z_all  = sort(cat([z_coarse, z_fine]))               # fine network renders these
Model (lego, 250 epochs)Network evals per rayTrain time / epochRender time / frameTest PSNR
Uniform, 128 samples1281.96 s156 ms24.51
Uniform, 256 samples2563.87 s540 ms24.52
Coarse 64 + fine 12864 + 1923.95 s337 ms24.74

Speed vs. quality. Simply doubling uniform samples from 128 to 256 costs 2× training time and 3.5× rendering time for no gain (+0.01 dB): at 128×128, 128 uniform samples already resolve this scene, and most extra samples land in empty space. At the same number of network evaluations, coarse/fine is the better use of compute (+0.22 dB over 256 uniform), because the fine samples concentrate around surfaces where the coarse weights are high. It also renders faster than 256 uniform, since the coarse pass is cheap. It is still about 2× slower than 128 uniform and has twice the parameters, and the visual difference at this resolution is small; the benefit should grow at higher resolution or with fewer coarse samples, where uniform sampling misses thin surfaces.

4.3 VolSDF 10 bonus

Implement an MLP that predicts signed distance and color (NeuralSurface), the eikonal loss, and VolSDF's SDF-to-density conversion. Explain what alpha and beta do, answer the three questions about beta, and comment on the settings that worked best.

Network. Positional encoding (6 frequencies) → 6-layer, 128-unit MLPWithInputSkips → a linear distance head with no output activation (distance can be negative). A 2-layer color MLP with a sigmoid reads the same features, so get_distance_color shares one forward pass.

# Eikonal: a true SDF has unit-norm gradient everywhere
eikonal_loss = ((gradients.norm(dim=-1) - 1) ** 2).mean()

# VolSDF density: sigma = alpha * Psi_beta(-d), Psi_beta = Laplace CDF (mean 0, scale beta)
half = 0.5 * exp(-|d| / beta)
density = alpha * where(d >= 0, half, 1 - half)

Sphere prior. With a randomly initialized distance network, training was unreliable: the zero level set starts in random places, so at β = 0.05 training collapsed with no surface at all, and at β = 0.01 a large spurious "floor" surface appeared. Following the geometric initialization idea in VolSDF/IDR, I predict distance as |x| - 1 plus a learned residual, so training starts from a sphere SDF with the correct sign everywhere. All results below use it.

Results (α = 10, eikonal weight 0.02, 128 samples, 250 epochs)

βTest PSNRGeometry
0.0121.53Sharpest: crisp base-plate edges and bucket
0.05 (default)21.53Good, slightly rounded edges
0.220.42Blobby, thickened, floating fragments near the arm

What α and β do

α is the density deep inside the object: Ψ goes to 1 for large negative distance, so density goes to α inside, to 0 outside, and equals α/2 exactly on the surface. It sets how opaque the solid is. β is the scale of the Laplace CDF, i.e. how wide the transition from empty to solid is around the zero level set. Small β makes density a near-step function of the SDF (a hard surface); large β spreads it into a soft shell around the surface.

1. How does high β bias the SDF? Low β?

High β lets the renderer explain the images with a thick, fuzzy shell of density, so the SDF doesn't need its zero level set in exactly the right place. The learned surface comes out smoothed and inflated, and stray density can float near the object, as in the blobby β = 0.2 mesh. Low β makes the rendered images depend almost entirely on where the SDF crosses zero, which biases the SDF toward a tight, sharp surface that sits exactly where the images say it should.

2. Easier to train with low or high β? Why?

High β. Density, and therefore gradient, is nonzero over a wide band around the surface, so many samples along every ray contribute signal even when the current surface is far from correct. With low β, only samples very close to the zero crossing get gradient, so a bad initial surface receives little signal to move. That is why the low-β runs without the sphere prior failed (collapse at 0.05, spurious floor at 0.01), while β = 0.2 still produced a coherent, if blobby, object.

3. More accurate surface with low or high β? Why?

Low β, as long as it trains. Density is concentrated at the zero level set, so the rendering loss directly supervises the surface location instead of a soft average around it. This matches the results: β = 0.01 gives the crispest geometry, with test PSNR equal to β = 0.05 and 1.1 dB better than β = 0.2.

Settings that worked best

β = 0.01 with the sphere prior, α = 10, and eikonal weight 0.02. The sphere prior removes the main disadvantage of low β (fragile early training) by starting from a valid SDF, and then low β gives the sharpest surface. The eikonal term keeps the network an actual distance function, which is what makes the β-controlled transition width meaningful in the first place.

5. 3D Gaussian Rasterization 35 pts

Implement projection of 3D Gaussians to 2D (5.1), 2D Gaussian evaluation (5.2), depth filtering and sorting (5.3), alpha and transmittance (5.4), and splatting (5.5), then render the pre-trained chair with render.py.

# 5.1  covariances and projected means
cov_3D = (R * s[:, None, :]) @ (R * s[:, None, :]).T       # R S S^T R^T  (isotropic: s^2 I)
cov_2D = J @ W @ cov_3D @ W.T @ J.T                        # W = camera.R
means_2D = camera.transform_points_screen(means_3D)[:, :2]

# 5.2  exponent of each 2D Gaussian at every pixel
power = -0.5 * einsum('npi,nij,npj->np', x - mu, inv(cov_2D), x - mu)

# 5.3  sort front-to-back, drop Gaussians behind the camera
z = camera.get_world_to_view_transform().transform_points(means)[:, 2]
idxs = argsort(z);  idxs = idxs[z[idxs] >= 0]

# 5.4  alpha and exclusive transmittance
alphas = opacities[:, None] * exp(power)
T = cumprod(cat([T_start, 1 - alphas]), dim=0)[:-1]

# 5.5  composite color, depth, and silhouette with the same weights
w = alphas * T
image = sum(colours * w, 0);  depth = sum(z * w, 0);  mask = sum(w, 0)

The mask is the total accumulated opacity, which is what render uses to blend in the background color. All four unit tests pass.

Turntable of the pre-trained chair: color, depth, and silhouette side by side
Figure 8. Pre-trained chair: color, depth, and mask (256×256, 32 views).

6. Training 3D Gaussian Representations 15 pts

Make the Gaussian parameters trainable, set up the optimizer with per-parameter learning rates, render and compute a loss in the training loop, and train isotropic Gaussians on the toy truck. Report the learning rates, number of iterations, PSNR, SSIM, and both GIFs.

Means, scales, colors, and opacities are trainable (quaternions are unused for isotropic Gaussians). The loss is L1 between the rendered and ground-truth images.

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colors0.01
Means0.0005

Iterations: 1000.   Mean PSNR: 30.010   Mean SSIM: 0.944

Means get the smallest learning rate: the point cloud initialization is already close to the surface, and large position steps scatter the Gaussians. Opacity goes through a sigmoid, so it tolerates a large step size; scales live in log space, where a small step already changes a Gaussian's size noticeably.

Training progress: rendered truck views (top) converging toward the ground-truth views (bottom)
Figure 9. Training progress (top: renders, bottom: ground truth).
Final renders of the trained truck from all training views
Figure 10. Final renders after training.

7.1 Rendering Using Spherical Harmonics 10 pts

Render the pre-trained chair using all of its spherical harmonic coefficients, attach the GIF next to the 5.5 GIF, and compare 2-3 matching views side by side.

Each Gaussian stores 16 SH coefficients per color channel (degree 3). For the direction from the camera center to each Gaussian, I evaluate the 16 real SH basis functions in one vectorized step and take the weighted sum:

sh = spherical_harmonics.view(N, 16, 3)
dirs = normalize(means_3D - camera.get_camera_center())
basis = stack([C0, -C1*y, C1*z, -C1*x, C2[0]*xy, ..., C3[6]*x*(xx - 3yy)], -1)   # (N, 16)
colours = clamp((basis[..., None] * sh).sum(1) + 0.5, min=0)

As a check, zeroing every coefficient except the DC term reproduces the 5.5 colors exactly.

Side-by-side comparisons (left: DC only, right: full SH)

7.2 Training on a Harder Scene 10 pts

Train Gaussians on the NeRF materials scene from random initialization. Compare a baseline using the Q6 setup against an improved approach (GIFs, PSNR, SSIM) and explain every modification.

Baseline: exactly the Q6 setup (same learning rates, L1 loss, isotropic Gaussians, 1000 iterations) with init_type="random". I then added improvements one at a time (each row keeps all the changes above it):

RunIterationsPSNRSSIM
Baseline (Q6 setup)100018.110.681
+ wider, fainter, larger initialization100022.740.831
+ means learning-rate decay100022.780.832
+ L1 + SSIM loss100022.890.845
+ anisotropic Gaussians100023.070.852
All of the above, trained longer500023.500.865

Modifications in the best run

  1. Initialization (+4.6 dB, by far the largest gain). The cameras sit 4.03 units from the origin with an NDC focal length of 3.125, so the visible region at the center is about ±1.3 units wide. The default random init (randn × 0.2) packs every Gaussian within about ±0.4, so spheres away from the center have no Gaussians nearby; in the baseline GIF they never appear. I spread the means with standard deviation 0.7, set initial opacity to 0.1 instead of about 1.0 (fully opaque blobs hide everything behind them and block gradients), and set the initial scale to 0.02 so each Gaussian covers about a pixel at this resolution.
  2. Means learning-rate schedule. 0.002, decaying exponentially to 0.00002 by the last iteration. Random points have to travel far early on, then settle.
  3. Loss: 0.8 · L1 + 0.2 · (1 − SSIM) (Gaussian window, 11×11, σ = 1.5), as in the 3DGS paper. This mainly helped SSIM (0.832 → 0.845).
  4. Anisotropic Gaussians: per-axis scales plus trainable rotations (quaternion learning rate 0.001), so Gaussians can stretch along the spheres' curved surfaces.
  5. Longer training: 5000 iterations instead of 1000 (+0.4 dB).

Other learning rates are unchanged from Q6 (opacities 0.05, scales 0.005, colors 0.01). I did not implement adaptive density control; the number of Gaussians stays at 10k.