16-825 Assignment 3: Neural Volume Rendering and Surface Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel grid
Pixel grid (vis_grid).
Ray directions
Ray directions (vis_rays).

1.4 Point Sampling

Sample points
Stratified sample points along the rays of the first camera.

1.5 Volume Rendering

Part 1 render
Spiral rendering of the box (part_1.gif).
Depth
Depth map, normalized by its maximum.

2. Optimizing a Basic Implicit Volume

Trained with --config-name=train_box (1000 epochs, 4096 random rays per step, MSE loss, lr 5e-4).

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Part 2 render
Spiral rendering of the optimized box (part_2.gif).

3. Optimizing a Neural Radiance Field (NeRF)

The MLP takes the harmonic (positional) encoding of the 3D position (6 frequencies), followed by 6 fully connected layers with 128 hidden units and the encoded input concatenated again at layer 3. A linear head with ReLU gives density, and a linear head with sigmoid gives color (no view dependence). Training: lego, 128×128 images, 250 epochs, 1024 rays per batch, 128 samples per ray in depth [2, 6], MSE loss, Adam with lr 5e-4 decayed by 0.8 every 50 epochs.

Part 3 NeRF
Spiral rendering of the trained NeRF (part_3.gif).

4.1 View Dependence

Density is predicted from the position features only. The color head receives the position features concatenated with a harmonic encoding of the normalized viewing direction (2 frequencies), passed through a 64-unit hidden layer with ReLU, then a sigmoid. Training: materials scene, 128×128 images, 250 epochs, 128 samples per ray in depth [1, 7], lr 1e-4 decayed by 0.8 every 50 epochs. Last batch MSE: 0.003158.

View-dependent NeRF on materials
View-dependent NeRF on the materials scene.

Trade-offs. Direction-dependent color can represent highlights and reflections, but adds flexibility that may overfit training views and reduce quality at unseen viewpoints. I limited the direction encoding to two frequencies and the color branch to 64 hidden units to constrain this flexibility. Generalization was not quantitatively evaluated.

4.2 Coarse/Fine Sampling

Two NeRF networks with the Part 3 architecture are trained jointly. The coarse network is evaluated at 64 stratified samples per ray (jittered within their intervals during training). Its volume-rendering weights define a piecewise-constant distribution along the ray, from which 64 more depths are drawn by inverse-CDF sampling. The fine network is evaluated at the sorted union of the 64 coarse and 64 new samples. The sampling does not backpropagate into the coarse weights. Each network has its own RGB MSE loss and the two are summed. Lego, 128×128 images, 250 epochs; the GIF below is rendered with the fine network after the final epoch (249).

Part 3
Part 3: single network, 128 uniform samples.
Part 4.2
Part 4.2: coarse/fine, 64 coarse + 128 fine samples.

Cost per ray: 64 coarse + 128 fine = 192 network evaluations (versus 128 in Part 3), and two networks to train and store.

Quality. Both saved animations reconstruct the bulldozer, tracks, raised arm, and base plate. In matched front, side, and rear views, the final coarse/fine rendering has a similar overall silhouette and recognizable details to Part 3. Both remain soft at fine edges and retain faint artifacts outside the base; there is no clear, consistent visual quality improvement at this resolution. The Part 3 GIF was saved at epoch 240 and the coarse/fine GIF at epoch 249, so this is a qualitative comparison rather than a controlled equal-budget benchmark.

Speed and memory. Hierarchical sampling concentrates the additional samples where coarse rendering weights are high, potentially spending fewer fine samples in empty regions. However, this setup uses 192 network evaluations per ray versus 128 in Part 3, a 50% increase in evaluation count, and stores two networks. This count is not a measured runtime ratio. No controlled end-to-end timing benchmark was recorded, so I do not claim a speedup. A weak coarse estimate may also miss thin structures and guide the fine network away from them.

5. 3D Gaussian Rasterization

Part 5 render
Pre-trained chair Gaussians rendered with my rasterizer (RGB, depth, mask).

6. Training 3D Gaussian Representations

Truck scene, isotropic Gaussians, L1 loss, Adam.

ParameterLearning rate
Opacities0.05
Scales0.005
Colours0.0025
Means0.00016

Iterations: 1000   PSNR: 29.415 dB   SSIM: 0.939

Part 6 training progress
Training progress: renders (top) and ground truth (bottom).
Part 6 final renders
Final renders.

7.1 Rendering Using Spherical Harmonics

Colors are computed from all degree-3 spherical-harmonic coefficients (16 per channel) and the direction from the camera center to each Gaussian, instead of only the DC term used in Part 5.

Part 5, DC only
Part 5: DC component only.
Part 7.1, full SH
Part 7.1: full spherical harmonics.

Side-by-side comparisons (left: DC only, right: spherical harmonics)

View 0 comparison
View 0.

In view 0, spherical harmonics produces darker green upholstery, especially across the backrest, and stronger contrast between the fabric and gold floral pattern. The chair outline is largely unchanged; the difference is primarily in its appearance.

View 8 comparison
View 8.

In view 8, the seat and backrest show darker green shading with spherical harmonics. The streaks around the legs and other poorly reconstructed regions remain in both images. Direction-dependent color changes the shading but does not repair these geometry artifacts.

View 16 comparison
View 16.

In view 16, the spherical-harmonic rendering again has darker, less uniformly bright upholstery, while the gold pattern remains visible. The silhouette and main structures are similar in both versions. The higher-order coefficients change appearance with the camera direction rather than changing the geometry.

7.2 Training on a Harder Scene

Materials scene, 10,000 randomly initialized isotropic Gaussians (init_type="random"), seed 1, 1000 iterations for each of three runs. Neither modification outperformed the baseline.

Baseline (Part 6 setup)First modificationRevised modification
Initial position std0.20.60.3
Initial scale / opacitydefault0.04 / 0.10.02 / 0.1
Initial LR opacities / scales0.05 / 0.0050.05 / 0.0050.05 / 0.005
Initial LR colours / means0.0025 / 0.000160.01 / 0.0010.0025 / 0.0005
LR scheduleconstantexponential decay to 10%exponential decay to 10%
LossL10.8 L1 + 0.2 local SSIM lossL1 for 100 iterations, then 0.9 L1 + 0.1 local SSIM loss
Color constraintunconstrainedunconstrainedclamp to [0, 1] after each update
PSNR (dB) ↑18.001−5.7236.742
SSIM ↑0.6600.1980.450

Baseline

Baseline training progress
Baseline training progress.
Baseline final renders
Baseline final renders.

First modification

First modification training progress
First modification training progress.
First modification final renders
First modification final renders.

Revised modification

Revised materials training progress
Revised training progress: predictions above ground truth.
Revised materials final renders
Final renders from the revised modification.

Motivation and results. The first modification spread the random initial points more widely, increased their initial scale, and lowered initial opacity to cover the multi-object scene with translucent Gaussians. Higher position/color learning rates were intended to allow faster adaptation; exponential decay was intended to stabilize later updates. The additional SSIM loss used 7×7 uniform windows to encourage local structural agreement. These changes did not improve the result. The reconstruction became excessively bright, and a validation-view inspection found RGB values as high as 4.12, outside the target range [0, 1].

The revision clamped optimized colors to [0, 1] after each update, reduced the initial position spread and scale, lowered the position/color learning rates, and delayed a smaller SSIM contribution until after 100 L1-only iterations. Bounding colors fixed the out-of-range RGB problem, and the revision improved over the first attempt from −5.723 to 6.742 dB PSNR and from 0.198 to 0.450 SSIM. Nevertheless, its renders remained overly bright and blurred, and both metrics were below baseline.

Best-performing method. The unmodified baseline achieved the best validation results: 18.001 dB PSNR and 0.660 SSIM. Neither proposed modification demonstrated an improvement. Because multiple settings changed together, these experiments cannot isolate the contribution of initialization, learning rates, scheduling, or SSIM. I report both unsuccessful attempts rather than describing them as improvements. A future controlled experiment would change one setting at a time while retaining the baseline setup.