Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Name: Yuqin Jiao  |  Andrew ID: sjiao2

A. Neural Volume Rendering (60 points)

1.3. Ray sampling (5 points)

Once you have implemented these methods, verify that your output matches the TA output by visualizing both xy_grid and rays with the vis_grid and vis_rays functions in the render_images function in volume_rendering_main.py.
Visualization of xy_grid

xy_grid

Visualization of rays

Rays

1.4. Point sampling (5 points)

Once you have done this, use the render_points method in render_functions.py in order to visualize the point samples from the first camera.
Point samples from the first camera

1.5. Volume rendering (20 points)

By default, your results will be written out to images/part_1.gif. Provide a visualization of the depth in your write-up. Note that the depth should be normalized by its maximum value.
Spiral rendering of part 1

Spiral rendering (part_1.gif)

Normalized depth of part 1

Depth (normalized by its maximum value)

2.1. Random ray sampling (5 points) and 2.2. Loss and training (5 points)

This will optimize the position and side lengths of a box, given a few ground truth images with known camera poses (in the data folder). Report the center of the box, and the side lengths of the box after training, rounded to the nearest 1/100 decimal place.
xyz
Center of the box0.250.25-0.00
Side lengths of the box2.021.521.51

2.3. Visualization

The code renders a spiral sequence of the optimized volume in images/part_2.gif. Compare this gif to the one below, and attach it in your write-up:
Spiral rendering of part 2

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

After training, a spiral rendering will be written to images/part_3.gif. Report your results.
Spiral rendering of the NeRF on the lego bulldozer

4. NeRF Extras (CHOOSE ONE! More than one is extra credit)

4.1 View Dependence (10 bonus points)

Add view dependence to your NeRF model! Specifically, make it so that emission can vary with viewing direction. Discuss the trade-offs between increased view dependence and generalization quality. While you may use the lego scene to test your code, please employ the materials scene to show the results of your method on your webpage (experimental settings can be found in nerf_materials.yaml and nerf_materials_highres.yaml).
Materials scene with view dependence

Materials scene with view dependence

Trade-offs between view dependence and generalization:

I made the predicted color depend on the viewing direction, allowing the model to represent view-dependent effects such as specular highlights and reflections. However, this also increases the model's flexibility, which may lead to overfitting to the training viewpoints and poorer generalization to unseen views.

B. 3D Gaussian Splatting (60 points)

5. 3D Gaussian Rasterization (35 points)

Submission: In your webpage, attach the GIF that you obtained by running render.py
Rendering of pre-trained 3D Gaussians

6 Training 3D Gaussian Representations (15 points)

Submission: In your webpage, include the following details: Learning rates that you used for each parameter. If you had experimented with multiple sets of learning rates, just mention the set that obtains the best performance in the next question. Number of iterations that you trained the model for. The PSNR and SSIM. Both the GIFs output by train.py.

Learning Rate Experiments:

I experimented with three different learning-rate configurations while keeping the other training settings consistent. Each experiment was trained for 1000 iterations using isotropic 3D Gaussians, an L1 reconstruction loss, and the Adam optimizer.

Experiment Means Colours Opacities Scales
A 0.00005 0.005 0.005 0.002
B 0.0001 0.01 0.01 0.005
C (Best) 0.0002 0.02 0.02 0.01

Best Configuration: Experiment C

Number of iterations: 1000

PSNR: 28.995 dB  |  SSIM: 0.933

Experiment C achieved the best PSNR and SSIM among the three configurations, indicating that the higher learning rates improved reconstruction quality within 1000 iterations.

Training progress

Training progress

Final renders after training

Final renders

7 Extensions (Choose at least one! More than one is extra credit)

7.1 Rendering Using Spherical Harmonics (10 points)

Submission: In your webpage, include the following details: Attach the GIF you obtained using render.py for questions 7.1 (this question) and 5.5 (older question). Attach 2 or 3 side by side RGB image comparisons of the renderings obtained from both the cases. The images that are being compared should correspond to the same view/frame. For each of the side by side comparisons that are attached, provide some explanation of differences (if any) that you notice.
Rendering without spherical harmonics (5.5)

Without spherical harmonics (5.5)

Rendering with spherical harmonics (7.1)

With spherical harmonics (7.1)

Comparison 1

Comparison 1 without spherical harmonics

Without spherical harmonics

Comparison 1 with spherical harmonics

With spherical harmonics

With spherical harmonics, the front view appears brighter and the green color is not as light as the image without harmonics. The depth and silhouette remain nearly same.

Comparison 2

Comparison 2 without spherical harmonics

Without spherical harmonics

Comparison 2 with spherical harmonics

With spherical harmonics

From this view, spherical harmonics produce more noticeable color variations on the backrest and seat, while the overall shape remains same.

Comparison 3

Comparison 3 without spherical harmonics

Without spherical harmonics

Comparison 3 with spherical harmonics

With spherical harmonics

With spherical harmonics, the front view appears brighter and the green color is not as light as the image without harmonics. The geometry and silhouette remain consistent.