Assignment 4: 3D Gaussian Splatting & Score Distillation Sampling

I extended the rasterizer to support spherical harmonics (SH) for view dependent color modeling. The implementation includes computing colors from all SH components (not just DC) based on viewing direction.

Summary: Across all views, the full spherical harmonics version consistently shows improved visual quality with better handling of highlights, shadows, and view dependent reflections. The view dependent shading makes the scene appear more realistic and three dimensional.

Techniques Used:

• Anisotropic Gaussians: Switched from isotropic to anisotropic Gaussians to better model complex geometry
• Gentler Learning Rate Decay: Used StepLR scheduler with gamma=0.8 (instead of 0.5) for more gradual learning rate reduction every 1000 iterations
• Increased SSIM Loss Weight: Adjusted loss to 0.4 * SSIM + 0.6 * MSE for better perceptual quality while maintaining structural fidelity
• Extended Training: Trained for 4000 iterations for better convergence

Implemented SDS loss following the DreamFusion paper, including both guided and unguided variants. The SDS loss uses a pretrained Stable Diffusion model to provide gradients that optimize latent representations toward matching text prompts.

Observations: Classifier free guidance significantly improves image quality and fidelity to the prompt. The guided versions show clearer details, better colors, and more recognizable objects. Without guidance, images tend to be more abstract and less detailed.

Using random camera sampling during training, the texture field (implemented as a ColorField MLP) is optimized to make the cow mesh match different text descriptions. Each view is rendered and optimized using SDS loss for 1000 iterations.

The latent_iter_ratio controls when the model switches from normal shading (warmup phase) to random shading (lambertian/textureless). Entropy and orientation regularization help maintain better geometry and reduce artifacts.

Implemented view dependent conditioning where different text embeddings are used based on camera azimuth:

• Front view (-60° to 60°): "front view of [prompt]"
• Side view (60° to 120°, -120° to -60°): "side view of [prompt]"
• Back view (120° to 180°, -180° to -120°): "back view of [prompt]"

This helps the model generate more consistent 3D geometry by providing directional context.

Comparison to Q2.3 Baseline: The view dependent conditioning produces more 3D consistent results with fewer multi face artifacts. Objects maintain better structural coherence across different viewing angles.

Implementation Details:

The pixel space SDS implementation follows this pipeline:

1. Encode RGB image to latents (required for U-Net compatibility)
2. Add noise to latents at timestep t
3. Predict noise using U-Net with text conditioning
4. Denoise using DDIM formula to get predicted clean latents: x₀ = (xₜ - σₜ * εₚᵣₑᵈ) / √αₜ
5. Decode predicted latents back to RGB space
6. Compute LPIPS perceptual loss between original and predicted RGB
7. Apply timestep weighting and backpropagate

This differs from standard latent space SDS by computing gradients in pixel space using a learned perceptual metric (LPIPS with VGG features) instead of simple L2 loss in latent space.

Training Configuration: Trained for 100 epochs (10,000 iterations total) with the same hyperparameters as Q2.3 (lambda_entropy: 0.01, lambda_orient: 0.01, latent_iter_ratio: 0.2).

Analysis:

Image Quality: The pixel space SDS results are noticeably worse than the latent space baseline (Q2.3). The generated geometry is less defined and the overall 3D structure is weaker.

Why the results are worse: While theoretically appealing, pixel space SDS with LPIPS has several disadvantages: (1) The additional VAE decode step introduces noise and errors into the gradient flow, (2) LPIPS perceptual loss may not provide gradients as well suited for 3D consistency as the simple L2 loss in latent space, (3) Pixel space gradients are noisier and less stable than latent space gradients, making optimization more difficult, (4) The diffusion model was trained in latent space, so computing loss in pixel space moves away from the model's natural operating domain.

Training Time: Slightly slower than latent space SDS due to the additional decode step and LPIPS computation.

Conclusion: Despite the intuitive appeal of using perceptual loss in pixel space, the standard latent space SDS approach produces superior results. The latent space representation appears to be better suited for providing clean, stable gradients for 3D generation tasks.