Assignment 3

Naveen Venkat (nvenkat)

Late Days: 0

1. Differentiable Volume Rendering

1.3. Ray sampling (10 points)

xy_grid	rays

python3 main.py --config-name=box

Images are written to results/1_3_rays.png and results/1_3_xy_vis.png.

1.4. Point sampling (10 points)

python3 main.py --config-name=box

The image is written to results/1_4_points.png.

1.5. Volume rendering (30 points)

Spiral Rendering	Depth map

python3 main.py --config-name=box

Images are written to images/part_1.gif and results/1_5_depth_map.png.

2. Optimizing a basic implicit volume

2.1. Random ray sampling (5 points)

Implemented in ray_utils.py.

2.2. Loss and training (5 points)

Implemented in main.py.

2.3. Visualization

The code renders a spiral sequence of the optimized volume in images/part_2.gif. Compare this gif to the one below, and attach it in your write-up:

python main.py --config-name=train_box

The image is written to images/part_2.gif.

3. Optimizing a Neural Radiance Field (NeRF) (30 points)

To run without view dependence, set the flag implicit_function.use_direction=False in configs/nerf_lego.yml.

python main.py --config-name=nerf_lego

No view dependence

The image is written to images/part_3no_dir.gif if not using view dependence. (See part 4.1 below for result with view dependence).

4. NeRF Extras (Choose at least one! More than one is extra credit)

4.1 View Dependence (10 pts)

To run with view dependence, set the flag implicit_function.use_direction=True in configs/nerf_lego.yml.

python main.py --config-name=nerf_lego

With view dependence

The image is written to images/part_3.gif.

After adding view dependence the following are observed:

1. I found more color variation in some key parts of the bulldozer, such as the top (roof) and one side (left side). Particularly (zoom in), the roof has more color variation (light yellow to light brown) as the camera is rotated around the scene. This range of color variation is not observed in the absence of view dependence (part 3 above).
2. Some spurious noise artefacts (seen at the bottom edge of the image in part 3 above) are mitigated. This indicates thatview dependence enables the model to capture richer information that helps the model generalize well. In some sense, view dependence gives meaning to camera viewpoint (via ray directions), which enables the network to be consistent across neighboring views, thereby avoiding spurious noise between adjacent frames. This demonstrates superior generalizability.

Caveat / Trade-off: Note that here we use a small RGB predictor head after the harmonic embedding of ray direction is introduced. This is essential to avoid overfitting. A deep RGB predictor (after adding view-dependence) could cause the model to memorize the image as observed from a certain viewpoint (implicitly captured by ray direction), hampering its generalizability. This gives rise to a trade-off between view-dependence and generalizability.

4.3 High Resolution Imagery (10 pts)

By default, the high resolution model renders at 400x400 resolution. Some settings used are as follows: batch_size=256, chunk_size=8192, 8-layered FC network with skip connection at index 4. Refer to config/nerf_lego_highres.yml for more.

python main.py --config-name=nerf_lego_highres

With view dependence, high resolution (400x400)