16-825 Assignment 3: Neural Volume Rendering and Surface Rendering
1. Differentiable Volume Rendering
1.3 Ray Sampling

vis_grid).
vis_rays).1.4 Point Sampling

1.5 Volume Rendering

part_1.gif).
2. Optimizing a Basic Implicit Volume
Trained with --config-name=train_box (1000 epochs, 4096 random rays per step, MSE loss, lr 5e-4).
| x | y | z | |
|---|---|---|---|
| Box center | 0.25 | 0.25 | 0.00 |
| Box side lengths | 2.01 | 1.50 | 1.50 |

part_2.gif).3. Optimizing a Neural Radiance Field (NeRF)
The MLP takes the harmonic (positional) encoding of the 3D position (6 frequencies), followed by 6 fully connected layers with 128 hidden units and the encoded input concatenated again at layer 3. A linear head with ReLU gives density, and a linear head with sigmoid gives color (no view dependence). Training: lego, 128×128 images, 250 epochs, 1024 rays per batch, 128 samples per ray in depth [2, 6], MSE loss, Adam with lr 5e-4 decayed by 0.8 every 50 epochs.

part_3.gif).4.1 View Dependence
Density is predicted from the position features only. The color head receives the position features concatenated with a harmonic encoding of the normalized viewing direction (2 frequencies), passed through a 64-unit hidden layer with ReLU, then a sigmoid. Training: materials scene, 128×128 images, 250 epochs, 128 samples per ray in depth [1, 7], lr 1e-4 decayed by 0.8 every 50 epochs. Last batch MSE: 0.003158.

Trade-offs. Direction-dependent color can represent highlights and reflections, but adds flexibility that may overfit training views and reduce quality at unseen viewpoints. I limited the direction encoding to two frequencies and the color branch to 64 hidden units to constrain this flexibility. Generalization was not quantitatively evaluated.
4.2 Coarse/Fine Sampling
Two NeRF networks with the Part 3 architecture are trained jointly. The coarse network is evaluated at 64 stratified samples per ray (jittered within their intervals during training). Its volume-rendering weights define a piecewise-constant distribution along the ray, from which 64 more depths are drawn by inverse-CDF sampling. The fine network is evaluated at the sorted union of the 64 coarse and 64 new samples. The sampling does not backpropagate into the coarse weights. Each network has its own RGB MSE loss and the two are summed. Lego, 128×128 images, 250 epochs; the GIF below is rendered with the fine network after the final epoch (249).


Cost per ray: 64 coarse + 128 fine = 192 network evaluations (versus 128 in Part 3), and two networks to train and store.
Quality. Both saved animations reconstruct the bulldozer, tracks, raised arm, and base plate. In matched front, side, and rear views, the final coarse/fine rendering has a similar overall silhouette and recognizable details to Part 3. Both remain soft at fine edges and retain faint artifacts outside the base; there is no clear, consistent visual quality improvement at this resolution. The Part 3 GIF was saved at epoch 240 and the coarse/fine GIF at epoch 249, so this is a qualitative comparison rather than a controlled equal-budget benchmark.
Speed and memory. Hierarchical sampling concentrates the additional samples where coarse rendering weights are high, potentially spending fewer fine samples in empty regions. However, this setup uses 192 network evaluations per ray versus 128 in Part 3, a 50% increase in evaluation count, and stores two networks. This count is not a measured runtime ratio. No controlled end-to-end timing benchmark was recorded, so I do not claim a speedup. A weak coarse estimate may also miss thin structures and guide the fine network away from them.
5. 3D Gaussian Rasterization

6. Training 3D Gaussian Representations
Truck scene, isotropic Gaussians, L1 loss, Adam.
| Parameter | Learning rate |
|---|---|
| Opacities | 0.05 |
| Scales | 0.005 |
| Colours | 0.0025 |
| Means | 0.00016 |
Iterations: 1000 PSNR: 29.415 dB SSIM: 0.939


7.1 Rendering Using Spherical Harmonics
Colors are computed from all degree-3 spherical-harmonic coefficients (16 per channel) and the direction from the camera center to each Gaussian, instead of only the DC term used in Part 5.


Side-by-side comparisons (left: DC only, right: spherical harmonics)

In view 0, spherical harmonics produces darker green upholstery, especially across the backrest, and stronger contrast between the fabric and gold floral pattern. The chair outline is largely unchanged; the difference is primarily in its appearance.

In view 8, the seat and backrest show darker green shading with spherical harmonics. The streaks around the legs and other poorly reconstructed regions remain in both images. Direction-dependent color changes the shading but does not repair these geometry artifacts.

In view 16, the spherical-harmonic rendering again has darker, less uniformly bright upholstery, while the gold pattern remains visible. The silhouette and main structures are similar in both versions. The higher-order coefficients change appearance with the camera direction rather than changing the geometry.
7.2 Training on a Harder Scene
Materials scene, 10,000 randomly initialized isotropic Gaussians (init_type="random"),
seed 1, 1000 iterations for each of three runs. Neither modification outperformed the baseline.
| Baseline (Part 6 setup) | First modification | Revised modification | |
|---|---|---|---|
| Initial position std | 0.2 | 0.6 | 0.3 |
| Initial scale / opacity | default | 0.04 / 0.1 | 0.02 / 0.1 |
| Initial LR opacities / scales | 0.05 / 0.005 | 0.05 / 0.005 | 0.05 / 0.005 |
| Initial LR colours / means | 0.0025 / 0.00016 | 0.01 / 0.001 | 0.0025 / 0.0005 |
| LR schedule | constant | exponential decay to 10% | exponential decay to 10% |
| Loss | L1 | 0.8 L1 + 0.2 local SSIM loss | L1 for 100 iterations, then 0.9 L1 + 0.1 local SSIM loss |
| Color constraint | unconstrained | unconstrained | clamp to [0, 1] after each update |
| PSNR (dB) ↑ | 18.001 | −5.723 | 6.742 |
| SSIM ↑ | 0.660 | 0.198 | 0.450 |
Baseline


First modification


Revised modification


Motivation and results. The first modification spread the random initial points more widely, increased their initial scale, and lowered initial opacity to cover the multi-object scene with translucent Gaussians. Higher position/color learning rates were intended to allow faster adaptation; exponential decay was intended to stabilize later updates. The additional SSIM loss used 7×7 uniform windows to encourage local structural agreement. These changes did not improve the result. The reconstruction became excessively bright, and a validation-view inspection found RGB values as high as 4.12, outside the target range [0, 1].
The revision clamped optimized colors to [0, 1] after each update, reduced the initial position spread and scale, lowered the position/color learning rates, and delayed a smaller SSIM contribution until after 100 L1-only iterations. Bounding colors fixed the out-of-range RGB problem, and the revision improved over the first attempt from −5.723 to 6.742 dB PSNR and from 0.198 to 0.450 SSIM. Nevertheless, its renders remained overly bright and blurred, and both metrics were below baseline.
Best-performing method. The unmodified baseline achieved the best validation results: 18.001 dB PSNR and 0.660 SSIM. Neither proposed modification demonstrated an improvement. Because multiple settings changed together, these experiments cannot isolate the contribution of initialization, learning rates, scheduling, or SSIM. I report both unsuccessful attempts rather than describing them as improvements. A future controlled experiment would change one setting at a time while retaining the baseline setup.