Assignment 3 — Neural Radiance Fields

0. Transmittance Calculation (10 points)

Compute the transmittance of a ray going through a non-homogeneous medium.

1. Differentiable Volume Rendering (30 points)

1.3. Ray Sampling (5 points)

Implement: get_pixels_from_image and get_rays_from_pixels in ray_utils.py.

1.4. Point Sampling (5 points)

Implement: StratifiedSampler.forward in sampler.py.

1.5. Volume Rendering (20 points)

Implement: VolumeRenderer._compute_weights, VolumeRenderer._aggregate, and depth rendering in renderer.py.

2. Optimizing a Basic Implicit Volume (10 points)

2.1. Random Ray Sampling (5 points)

Implement: get_random_pixels_from_image in ray_utils.py.

2.2. Loss and Training (5 points)

Implement: MSE loss in the train function in volume_rendering_main.py.

Training Progress

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

Implement: NeuralRadianceField MLP in implicit.py and loss in train_nerf.

View-Independent NeRF

Basic NeRF implementation without view dependence, trained on the lego bulldozer dataset.

4. NeRF Extras (10 points)

4.1. View Dependence (10 points)

Extended NeRF to handle view-dependent effects. The network now takes viewing direction as input, allowing it to model specular highlights and reflections on the lego pieces.

Analysis

The view-dependent NeRF produces more photorealistic results with accurate specular reflections, especially visible on the metallic and glossy surfaces of the bulldozer. However, there's a trade-off: the increased capacity can lead to overfitting on training views if not properly regularized. The view-independent variant generalizes better to novel views but lacks realistic reflections.

4.2. Coarse/Fine Sampling (Extra Credit)

[Optional: Implement hierarchical sampling with coarse and fine networks. Discuss speed/quality trade-offs.]

5. Sphere Tracing (10 points)

Implement: sphere_tracing function in renderer.py.

Implementation notes: Sphere tracing iteratively steps along each ray by the distance to the nearest surface (given by the SDF). This allows efficient rendering of implicit surfaces without uniform sampling. The algorithm terminates when either the ray gets very close to the surface (intersection found) or exceeds the maximum number of steps.

6. Optimizing a Neural SDF (15 points)

Implement: NeuralSurface.get_distance in implicit.py and eikonal_loss in losses.py.

7. VolSDF (15 points)

Implement: NeuralSurface.get_distance_color, get_color in implicit.py and sdf_to_density in renderer.py.

SDF to Density Conversion

The VolSDF paper introduces parameters α (alpha) and β (beta) for converting SDF to density:

• β (beta): Controls the "sharpness" of the transition from empty space to surface. High β creates a sharper, more concentrated density around the zero level set, while low β spreads the density over a wider region.
• α (alpha): Acts as a learned scaling factor that adapts during training.

Analysis Questions

1. High vs Low β bias: High β biases the SDF to be very sharp near surfaces, creating a thin shell of density. Low β allows more diffuse density distributions.
2. Training difficulty: Low β is easier to train initially because gradients flow through a larger region. High β requires more precise SDF predictions early in training.
3. Surface accuracy: High β leads to more accurate surfaces once converged, as it enforces tighter constraints on where density exists. Low β may result in "bloated" surfaces.

Hyperparameter Settings

Best results achieved with: alpha: 50.0, beta: 0.1. By increasing beta, the model produced sharper surfaces with better-defined geometry, while a moderate alpha helped maintain stable training dynamics.

8. Neural Surface Extras (10 points)

8.1. Render a Large Scene with Sphere Tracing (10 points)

Complex scene composed subtraction of multiple primitives. This shape, which has cube-like holes inside a sphere, shows how sphere tracing can be efficient, but can fail when some of the features can't be reached due to occlusion, such as the inner cube incut here.

Conclusion & Reflections

This assignment explored both volume-based and surface-based neural 3D representations:

• Volume Rendering: Differentiable rendering enables learning 3D structure from 2D supervision. NeRF's success relies heavily on positional encoding and sufficient sample density.
• Surface Rendering: SDFs provide better geometric priors and can be more sample-efficient. Sphere tracing is elegant for rendering implicit surfaces.
• Trade-offs: NeRF excels at view-dependent effects but requires many training views. VolSDF produces cleaner geometry and can work with fewer views, but is more complex to implement.