Homework 3

Jacob Lessing

A. Neural Volume Rendering

1. Differential Volume Rendering

1.3 Ray Sampling

1.4 Point Sampling

1.5 Volume Rendering

2. Optimizing a basic implicit volume

2.2 Loss and Training

Box center: (0.25, 0.25, 0.00) Box side lengths: (2.01, 1.50, 1.50)

2.3 Visualization

3. Optimizing a Neural Radiance Field (NeRF)

4. NeRF Extras

4.1 View Dependence

With sufficiently large view dependence, the NeRF can drive loss down without learning a coherent global scene structure. The NeRF simply cheats by leaning on the view-dependent component of the network to learn each view semi-independently, which greatly compromises generalization quality because without learning a physically-consistent object, the network has nothing coherent off of which to base general views. With that said, view dependence gets us direction-dependent light reflections off specular objects. To navigate this tradeoff, I used some priors. Firstly, density should not depend on view, so view direction only enters the network after density is calculated. Further, direction dependence only gets a low-frequency (2 harmonic) embedding which forces direction dependence to have relatively slow-shifting effects. The current implementation results in a NeRF where direction-dependent light reflections are visible and the overall object geometry and color are consistent.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

6. Training 3D Gaussian Representations

Hyperparameters: opacities lr: 0.05, pre_act_scales lr: 0.05, colours lr: 0.005, means lr: 0.0005, num iterations: 1000

Training Progress:

Evaluation: Mean PSNR: 29.959, Mean SSIM: 0.943

7. Extensions

7.1 Rendering Using Spherical Harmonics

Rendered Gifs

Without SH (5.5) With SH (7.1)

Side by Side frame comparison

Frame Without SH With SH Analysis
00 The render without SH appears somewhat flat, whereas the render with SH shows significantly sharper detail to the shadows & lighting, which makes sense because apparent light reflection is a view-dependent effect
25 I don’t see SHs making any difference in the visualization of the backside of the chair. Perhaps there were few training views of the back, so there wasnt enough variety to learn significant view-dependent effects