16825 - HW4

1.1 3D Gaussian Rasterization

1.2 Training 3D Gaussian Representations

I used lr = 0.001 for all parameters, and trained for 1000 iters. I obtained PSNR and SSIM of 28.630 and 0.924.

1.3 Training with Spatial Harmonics

I modified the code to enable harmonics training. Below are the isotrophic(left) implementation and the spatial harmonics(right) implementation.

From the gif, we can see spatial harmonics allows for modeling higher-order effects like reflection, especially visible by the texture change on the surface of the chair. We provide some more side-by-side comparisons below (index 0, 12, 18):

1.3 Training with Harder Scenes

I first repeated training with the same setting as 1.2, but observed bad optimization behavior. As mentioned in the readme, since the initialization is centralized as a gaussian, the network had hard time fitting to distinct objects. To try to solve this, I added a cosine learning rate scheduler. Since gaussians also need to move significantly to individual objects in the scene, I increased the lr for the means of the gaussians to 0.05. I later tuned other learning rates to 0.01 for faster convergence. Reference (PSNR: 16.543, SSIM: 0.543):

Adjusted LR (PSNR: 19.971, SSIM: 0.446):

With these adjustments, the model was able to better fit the more complex scene. However, the SSIM is still worse than the baseline. This could be from the learning rate for the colors being too high, or not using the SSIM loss.

SDS Loss + Image Optimization

Prompt: a hamburger

Prompt: a standing corgi dog

Prompt: a cow

Prompt: a rocket

Texture Map Optimization for Mesh

Prompt: a cartoon cow

Prompt: a neon cow

(kind of scary)