16-825 Learning for 3D Vision: Assignment 4
16-825, Fall 2026 · Zijin Ye (Andrew ID: zijiny)
3D Gaussian Splatting and Diffusion-guided Optimization. All results below were produced with the
submitted code on a Google Colab A100 GPU.
1.1 3D Gaussian Rasterization
The rasterizer projects each 3D Gaussian to a 2D Gaussian (Σ2D = J W Σ3D WT JT), sorts the Gaussians by camera-space depth after dropping those behind the camera, and alpha-composites them front to back. Colour, depth and silhouette all use the same weights αiTi: C = Σ ciαiTi, D = Σ ziαiTi, M = Σ αiTi. Everything is vectorized over Gaussians and pixels, and the Gaussians are splatted in mini-batches that carry the transmittance over from one batch to the next.
render.py
(left to right: RGB, depth, silhouette).X_view = X_world · R + T), so the world-to-camera rotation in the column-vector
formula above is W = RT, and that is what compute_cov_2D uses.
With it, tests 1 and 3 pass. The expected values in tests 2 and 4 match W = R
exactly; I verified this by recomputing them. That convention produces visibly wrong
Gaussian orientations, as the comparison below shows, so I kept W = RT.
1.2 Training 3D Gaussian Representations
Isotropic Gaussians are initialized from the provided truck point cloud and optimized with Adam, using a separate learning rate for each parameter and an L1 photometric loss.
| Parameter | Learning rate |
|---|---|
| Opacities (pre-sigmoid) | 0.05 |
| Scales (log) | 0.005 |
| Colours | 0.02 |
| Means | 0.0005 |
| Iterations | Loss | Test PSNR | Test SSIM |
|---|---|---|---|
| 1000 | L1 | 29.824 | 0.942 |
1.3.1 Rendering Using Spherical Harmonics
The colour of each Gaussian is evaluated from all 16 SH coefficients per channel (degrees 0–3),
using a vectorized version of get_color. The direction for each Gaussian is the
normalized vector from the camera centre to its mean. render.py --use_sh switches
between the two modes, so both GIFs come from the same code.
Side-by-side comparisons (left: DC only, right: with SH)
1.3.2 Training on a Harder Scene
Both setups are in Q1/train_harder_scene.py and are trained on the NeRF-synthetic
materials scene at 128×128 with init_type="random":
python train_harder_scene.py --baseline --num_itrs 1000 --viz_freq 40 # baseline (Q1.2 setup)
python train_harder_scene.py --num_itrs 5000 --viz_freq 200 # improved approach
Baseline. Identical to Q1.2: 10,000 isotropic Gaussians with the default random init (means ~ N(0, 0.2²)), the same learning rates, L1 loss and 1000 iterations.
Improved approach. Changes relative to Q1.2:
- Anisotropic Gaussians: per-axis scales and trainable quaternions (lr 0.001), so elongated or flat structure doesn't need many tiny isotropic blobs.
- Scene-covering initialization: still random, but the means are sampled uniformly inside a ball of radius 1.3, the extent of the spheres, instead of from N(0, 0.2²), which puts every Gaussian in a small blob at the origin. Initial scales come from the nearest-neighbour spacing of the samples, and the initial opacity is 0.1.
- Loss: 0.8·L1 + 0.2·(1 − SSIM), as in the original paper, with a differentiable SSIM using an 11×11 Gaussian window.
- Learning-rate schedule: the means' learning rate decays exponentially from 1.6e-3 to 1.6e-5 over training, for large moves early and fine refinement late. Other attributes keep constant rates (opacities 0.05, scales 0.005, colours 0.01).
- Simplified adaptive density control: every 100 iterations between iterations 500 and 3000, Gaussians with a large average positional gradient are densified. Small ones are cloned; large ones are split into two samples drawn from the Gaussian at 1/1.6 of its scale. Nearly transparent or oversized Gaussians are pruned, with a cap of 40,000. The Adam state is carried over for surviving Gaussians.
- Longer training: 5000 iterations.
- Tiled splatting (
Scene.splat_tiled): each Gaussian is evaluated only on the 16×16 tiles its footprint can reach, within radius √(2 ln(255·o) λmax), beyond which α is already clamped to zero. The result is identical to dense splatting, but it keeps 40k Gaussians affordable. Both runs together, including the dataset download, took 7.4 minutes on an A100.
| Method | Iterations | Final #Gaussians | Test PSNR | Test SSIM |
|---|---|---|---|---|
| Baseline (Q1.2 setup, random init) | 1000 | 10,000 | 18.412 | 0.677 |
| Improved | 5000 | 39,920 | 25.130 | 0.911 |
Comparison. The baseline never recovers from its initialization. All Gaussians start in a small blob at the origin, and with isotropic Gaussians, a small learning rate for the means and no way to add Gaussians, they only spread a little. The result is a blurry cluster in the middle of the frame, and the outer spheres are never reconstructed. The improved approach starts with Gaussians covering the whole scene, lets densification grow them from 10k to about 40k where the gradients ask for detail, and gets the specular highlights and the different materials (chrome, gold, red, green, matte) right, for +6.7 dB PSNR.





2.1 SDS Loss and Image Optimization
sds_loss samples t ~ U(0.02, 0.98)·T, noises the latents with the DDIM scheduler,
and predicts the noise with the frozen UNet. With guidance, it also predicts the
unconditional noise from the empty-prompt embedding and combines them as
ε̂ = εu + s(εc − εu) with s = 100. The gradient
w(t)(ε̂ − ε), with w(t) = 1 − ᾱt, is applied through the reparameterization
0.5·‖latents − (latents − grad).detach()‖², whose derivative with respect to the
latents is exactly that gradient. The UNet Jacobian is never computed.
Every run optimized the 64×64×4 latent for 2000 iterations. Without guidance, the image converges to a blurry, nearly uniform blob that gets flatter with more iterations, so I show iteration 400, matching the reference. With guidance, I show the final iteration 2000. On an A100 each prompt took about 92 s without guidance and 174 s with it.








2.2 Texture Map Optimization for Mesh
The cow's geometry is fixed, and a ColorField MLP predicts the colour of each
vertex. At each of the 2000 iterations I sample a random camera at distance 3 with azimuth
U(0°, 360°) and elevation U(−10°, 40°), put a point light at the camera centre plus a small
random offset, render at 512×512 with the soft rasterizer, encode the render with the VAE,
and apply SDS with guidance. Each prompt took about 6 minutes on an A100.



2.3 NeRF Optimization
Each step renders the NeRF at 64×64 from a random camera, bilinearly upsamples the render to 512×512, encodes it with the VAE and applies guided SDS. Final settings:
| Hyperparameter | Value |
|---|---|
lambda_entropy | 1e-3 |
lambda_orient | 1e-2 |
latent_iter_ratio | 0.2 (normal shading for the first 20% of iterations, then random textureless/Lambertian) |
| Iterations | 10,000 (about 40 min per prompt on an A100) |
Each GIF is the 100-frame test turntable (left: RGB, right: depth), rendered at the training resolution of 64×64 and shown scaled up. The background is the last random colour sampled during training, which the provided test loop reuses. A still frame strip under each GIF shows 8 evenly spaced views (top: RGB, bottom: depth).
"a standing corgi dog"


"a hamburger"


"a yellow rubber duck"


2.4.1 View-dependent Text Embedding
With --view_dep_text 1, prepare_embeddings also encodes
"<prompt>, front view", "<prompt>, side view" and "<prompt>, back view". At each
step the conditional embedding is interpolated using the camera azimuth relative to the
default front view: for |az| < 90°, it blends front and side with weight 1 − |az|/90; otherwise
it blends side and back with weight 1 − (|az| − 90)/90. I also fixed a bug in the provided
prepare_embeddings: it put the prompt list's string representation into the text
("['a standing corgi dog'], front view") instead of the prompt itself.
All other settings are the same as in 2.3: same hyperparameters, seed and number of iterations.
"a standing corgi dog" with view-dependent text


"a yellow rubber duck" with view-dependent text


Analysis
- When it helps (corgi). The "side view" and "back view" prompts give SDS a target that matches the actual camera, so views no longer all pull toward the canonical front-facing dog. The result is a more plausible 3D shape, an elongated body in profile, and a back view without a face.
- When it does not (duck). View-dependent text is only a soft prior, and it works only as far as Stable Diffusion 2.1 understands "front/side/back view" for that object. Corgis appear in many poses in the training data. A rubber duck is almost always photographed front-on or in 3/4 view, so "back view" still produces faces and beaks, and the interpolated embeddings near ±90° and 180° do not agree on where the head is. With a guidance scale of 100, those conflicting per-view targets are strong enough to create several heads.
- Takeaway. View-dependent prompting reduces the Janus problem for objects whose different views are well represented in the diffusion model, but it is not a reliable fix on its own. Stronger remedies such as Perp-Neg, multi-view diffusion priors or lower guidance would be needed for objects like the duck.
2.4.3 Variation of the SDS Loss: Pixel Space
Implementation (SDS.sds_loss_pixel, enabled with
Q23_nerf_optimization.py --sds_space pixel). The rendered image x is encoded and noised
exactly as before, and the UNet predicts the noise with classifier-free guidance (s = 100). Instead
of using ε̂ − ε as a latent gradient, the prediction gives a one-step estimate of the clean latent,
x̂0 = (zt − √(1−ᾱt) ε̂) / √ᾱt, which is decoded with the VAE
to a target image D(x̂0). The loss is a weighted pixel-space reconstruction error,
Lpixel = ½ · wpix(t) · ‖x − sg(D(x̂0))‖², wpix(t) = √(ᾱt(1 − ᾱt)),
so ∂L/∂x = wpix(t)(x − D(x̂0)) is computed directly in pixel space. Encoding, denoising and decoding all run without gradients, so nothing is backpropagated through the VAE encoder. The weight is chosen so that the loss is the pixel-space counterpart of latent SDS: since z0 − x̂0 = √((1−ᾱ)/ᾱ) (ε̂ − ε), the latent SDS gradient (1−ᾱ)(ε̂ − ε) equals √(ᾱ(1−ᾱ)) (z0 − x̂0). The two losses have very different magnitudes, and the entropy and orientation regularizers have fixed weights. So for the first 50 steps I scale the pixel loss to give the same gradient norm with respect to the NeRF render as latent SDS (median ratio: 10), and keep that scale for the rest of training. Everything else matches the Q2.3 corgi run: hyperparameters, seed and 10,000 iterations.


| SDS variant | Optimization time (10k iters, A100) | Result |
|---|---|---|
| Latent space (Q2.3) | 2357 s | Sharp, saturated corgi with plausible geometry |
| Pixel space (L2) | 2672 s (+13%) | Small, desaturated corgi; frontal from every view; hazy depth |
Analysis
- Image quality: worse. The pixel-space corgi is recognizable but small and washed out, grey-blue instead of orange and white, and its colours bleed into the background. It shows the same frontal face from every azimuth, like a billboard, and the depth map has a haze of low density around the object instead of a clean silhouette.
- Why. (1) With guidance scale 100, ε̂ is strongly extrapolated, so the one-step estimate x̂0 is far outside the latent distribution. In latent space this does no harm, because the update is linear in ε̂ − ε. In pixel space, the extrapolated latent goes through the nonlinear decoder and is clamped to [0, 1], which destroys much of the direction information and leaves a noisy, saturated target. (2) At large t, x̂0 is close to the posterior mean, a blurry average image. An L2 loss toward a blurry target pulls the render toward average colours, which explains the desaturation, and weakens the shape signal that would make side views look different. (3) Plain L2 treats every pixel equally, so the large uniform background dominates the gradient. That favours a small object and lets low-density haze in the background survive.
- Training time: 13% slower. Each step skips backpropagation through the VAE encoder, but adds a full VAE decode of x̂0 at 512×512, which costs more. The 50 calibration steps also compute both losses.
- Possible improvements. A perceptual loss such as LPIPS instead of L2, a lower guidance scale or multi-step denoising for a cleaner x̂0, or combining a small pixel term with latent SDS, as HiFA does, rather than replacing it.