16-825 Learning for 3D Vision: Assignment 4

16-825, Fall 2026 · Zijin Ye (Andrew ID: zijiny)
3D Gaussian Splatting and Diffusion-guided Optimization. All results below were produced with the submitted code on a Google Colab A100 GPU.

1.1 3D Gaussian Rasterization

The rasterizer projects each 3D Gaussian to a 2D Gaussian (Σ2D = J W Σ3D WT JT), sorts the Gaussians by camera-space depth after dropping those behind the camera, and alpha-composites them front to back. Colour, depth and silhouette all use the same weights αiTi: C = Σ ciαiTi, D = Σ ziαiTi, M = Σ αiTi. Everything is vectorized over Gaussians and pixels, and the Gaussians are splatted in mini-batches that carry the transmittance over from one batch to the next.

Rendered GIF of the pre-trained chair: RGB, depth and silhouette
1.1.5: pre-trained chair Gaussians rendered with render.py (left to right: RGB, depth, silhouette).
Note on the unit tests. PyTorch3D cameras use row vectors (X_view = X_world · R + T), so the world-to-camera rotation in the column-vector formula above is W = RT, and that is what compute_cov_2D uses. With it, tests 1 and 3 pass. The expected values in tests 2 and 4 match W = R exactly; I verified this by recomputing them. That convention produces visibly wrong Gaussian orientations, as the comparison below shows, so I kept W = RT.
Same chair view rendered with W = R transpose (left, sharp) and W = R (right, streaked)
The same view rendered with W = RT (left) and W = R (right). With W = R, the misoriented Gaussians smear along the wrong directions.

1.2 Training 3D Gaussian Representations

Isotropic Gaussians are initialized from the provided truck point cloud and optimized with Adam, using a separate learning rate for each parameter and an L1 photometric loss.

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.02
Means0.0005
IterationsLossTest PSNRTest SSIM
1000L129.8240.942
Training progress GIF: predictions on top, ground truth below
Training progress (top: current renders, bottom: ground truth).
Final renders of the trained truck from the training views
Final renders of all training views after training.

1.3.1 Rendering Using Spherical Harmonics

The colour of each Gaussian is evaluated from all 16 SH coefficients per channel (degrees 0–3), using a vectorized version of get_color. The direction for each Gaussian is the normalized vector from the camera centre to its mean. render.py --use_sh switches between the two modes, so both GIFs come from the same code.

Chair render using only the DC colour
1.1.5: DC (view-independent) colour only.
Chair render using all spherical harmonic components
1.3.1: full spherical harmonics (view-dependent).

Side-by-side comparisons (left: DC only, right: with SH)

View 12: DC on the left, SH on the right
View 12. With SH, the seat cushion and backrest are a deeper green and the gold brocade stands out more. The DC-only render is uniformly bright and looks washed out, because it omits the higher-order terms that darken the fabric at this viewing angle.
View 16: DC on the left, SH on the right
View 16 (front). The SH render shows the sheen of the silk: the centre of the backrest is darker while the inner sides catch more light, so the chair looks shaded and three-dimensional. The DC render looks flat.
View 27: DC on the left, SH on the right
View 27 (underside). The two are nearly identical. The matte beige wood is close to Lambertian, so its higher-order SH coefficients are small and the colour barely changes with viewing direction.

1.3.2 Training on a Harder Scene

Both setups are in Q1/train_harder_scene.py and are trained on the NeRF-synthetic materials scene at 128×128 with init_type="random":

python train_harder_scene.py --baseline --num_itrs 1000 --viz_freq 40   # baseline (Q1.2 setup)
python train_harder_scene.py --num_itrs 5000 --viz_freq 200             # improved approach

Baseline. Identical to Q1.2: 10,000 isotropic Gaussians with the default random init (means ~ N(0, 0.2²)), the same learning rates, L1 loss and 1000 iterations.

Improved approach. Changes relative to Q1.2:

MethodIterationsFinal #GaussiansTest PSNRTest SSIM
Baseline (Q1.2 setup, random init)100010,00018.4120.677
Improved500039,92025.1300.911

Comparison. The baseline never recovers from its initialization. All Gaussians start in a small blob at the origin, and with isotropic Gaussians, a small learning rate for the means and no way to add Gaussians, they only spread a little. The result is a blurry cluster in the middle of the frame, and the outer spheres are never reconstructed. The improved approach starts with Gaussians covering the whole scene, lets densification grow them from 10k to about 40k where the gradients ask for detail, and gets the specular highlights and the different materials (chrome, gold, red, green, matte) right, for +6.7 dB PSNR.

Baseline training progress on the materials scene
Baseline training progress over 1000 iterations (top: renders, bottom: ground truth; one frame per 40 iterations)
Improved training progress on the materials scene
Improved training progress over 5000 iterations (one frame per 200 iterations)
Baseline final renders
Baseline final renders
Improved final renders
Improved final renders
Eight training views: baseline (top) and improved (bottom)
The same eight training views. Top: baseline. Bottom: improved.

2.1 SDS Loss and Image Optimization

sds_loss samples t ~ U(0.02, 0.98)·T, noises the latents with the DDIM scheduler, and predicts the noise with the frozen UNet. With guidance, it also predicts the unconditional noise from the empty-prompt embedding and combines them as ε̂ = εu + s(εc − εu) with s = 100. The gradient w(t)(ε̂ − ε), with w(t) = 1 − ᾱt, is applied through the reparameterization 0.5·‖latents − (latents − grad).detach()‖², whose derivative with respect to the latents is exactly that gradient. The UNet Jacobian is never computed.

Every run optimized the 64×64×4 latent for 2000 iterations. Without guidance, the image converges to a blurry, nearly uniform blob that gets flatter with more iterations, so I show iteration 400, matching the reference. With guidance, I show the final iteration 2000. On an A100 each prompt took about 92 s without guidance and 174 s with it.

a hamburger, no guidance
"a hamburger": no guidance, 400 iters
a hamburger, guidance
"a hamburger": guidance, 2000 iters
a standing corgi dog, no guidance
"a standing corgi dog": no guidance, 400 iters
a standing corgi dog, guidance
"a standing corgi dog": guidance, 2000 iters
a bowl of ramen, no guidance
"a bowl of ramen": no guidance, 400 iters
a bowl of ramen, guidance
"a bowl of ramen": guidance, 2000 iters
a red sports car, no guidance
"a red sports car": no guidance, 400 iters
a red sports car, guidance
"a red sports car": guidance, 2000 iters

2.2 Texture Map Optimization for Mesh

The cow's geometry is fixed, and a ColorField MLP predicts the colour of each vertex. At each of the 2000 iterations I sample a random camera at distance 3 with azimuth U(0°, 360°) and elevation U(−10°, 40°), put a point light at the camera centre plus a small random offset, render at 512×512 with the soft rasterizer, encode the render with the VAE, and apply SDS with guidance. Each prompt took about 6 minutes on an A100.

Initial random texture
Initial (random) texture
a dotted black and white cow
"a dotted black and white cow"
an orange golden bull
"an orange golden bull"

2.3 NeRF Optimization

Each step renders the NeRF at 64×64 from a random camera, bilinearly upsamples the render to 512×512, encodes it with the VAE and applies guided SDS. Final settings:

HyperparameterValue
lambda_entropy1e-3
lambda_orient1e-2
latent_iter_ratio0.2 (normal shading for the first 20% of iterations, then random textureless/Lambertian)
Iterations10,000 (about 40 min per prompt on an A100)

Each GIF is the 100-frame test turntable (left: RGB, right: depth), rendered at the training resolution of 64×64 and shown scaled up. The background is the last random colour sampled during training, which the provided test loop reuses. A still frame strip under each GIF shows 8 evenly spaced views (top: RGB, bottom: depth).

"a standing corgi dog"

Corgi NeRF turntable: RGB and depth
Turntable (left: RGB, right: depth)
Eight views of the corgi NeRF, RGB and depth
A clearly recognizable corgi with a consistent orange-and-white coat and a clean, solid depth map. The head tends to turn toward the camera in most views, an early sign of the multi-face problem examined in 2.4.1.

"a hamburger"

Hamburger NeRF turntable: RGB and depth
Turntable (left: RGB, right: depth)
Eight views of the hamburger NeRF, RGB and depth
Bun, patty and lettuce layers are all present, and the depth shows a stacked, rounded shape. The object is nearly rotationally symmetric, so every view looks like a valid hamburger and there is no view-consistency problem.

"a yellow rubber duck"

Rubber duck NeRF turntable: RGB and depth
Turntable (left: RGB, right: depth)
Eight views of the duck NeRF, RGB and depth
The front and side views are a convincing rubber duck with a correct beak profile. From behind (4th–5th frames), however, a second face with a beak appears where the tail should be. This is the Janus (multi-face) artifact: SDS optimizes each view independently with the same prompt, and the diffusion model mostly produces front-facing ducks.

2.4.1 View-dependent Text Embedding

With --view_dep_text 1, prepare_embeddings also encodes "<prompt>, front view", "<prompt>, side view" and "<prompt>, back view". At each step the conditional embedding is interpolated using the camera azimuth relative to the default front view: for |az| < 90°, it blends front and side with weight 1 − |az|/90; otherwise it blends side and back with weight 1 − (|az| − 90)/90. I also fixed a bug in the provided prepare_embeddings: it put the prompt list's string representation into the text ("['a standing corgi dog'], front view") instead of the prompt itself.

All other settings are the same as in 2.3: same hyperparameters, seed and number of iterations.

"a standing corgi dog" with view-dependent text

View-dependent corgi NeRF turntable: RGB and depth
View-dependent turntable (left: RGB, right: depth)
Corgi: 2.3 RGB, 2.4.1 RGB and 2.4.1 depth over eight views
Top: 2.3 (single prompt). Middle: 2.4.1 RGB. Bottom: 2.4.1 depth. With view-dependent conditioning, the side views show a proper corgi build, a long low body on short legs, and the depth confirms the elongated shape. The face appears only in front and side views, and the back view shows just the back of the head, the ears and the rear. Without it, the body is more compact and the head keeps turning toward the camera.

"a yellow rubber duck" with view-dependent text

View-dependent duck NeRF turntable: RGB and depth
View-dependent turntable (left: RGB, right: depth)
Duck: 2.3 RGB, 2.4.1 RGB and 2.4.1 depth over eight views
Top: 2.3 (single prompt). Middle: 2.4.1 RGB. Bottom: 2.4.1 depth. Here view-dependent conditioning did not remove the Janus artifact; it rearranged it. The 0° ("front") view shows a side profile with a beak on each side, front-facing faces appear at two other azimuths, and the depth map has bumps on both sides of the head.

Analysis

2.4.3 Variation of the SDS Loss: Pixel Space

Implementation (SDS.sds_loss_pixel, enabled with Q23_nerf_optimization.py --sds_space pixel). The rendered image x is encoded and noised exactly as before, and the UNet predicts the noise with classifier-free guidance (s = 100). Instead of using ε̂ − ε as a latent gradient, the prediction gives a one-step estimate of the clean latent, x̂0 = (zt − √(1−ᾱt) ε̂) / √ᾱt, which is decoded with the VAE to a target image D(x̂0). The loss is a weighted pixel-space reconstruction error,

Lpixel = ½ · wpix(t) · ‖x − sg(D(x̂0))‖²,   wpix(t) = √(ᾱt(1 − ᾱt)),

so ∂L/∂x = wpix(t)(x − D(x̂0)) is computed directly in pixel space. Encoding, denoising and decoding all run without gradients, so nothing is backpropagated through the VAE encoder. The weight is chosen so that the loss is the pixel-space counterpart of latent SDS: since z0 − x̂0 = √((1−ᾱ)/ᾱ) (ε̂ − ε), the latent SDS gradient (1−ᾱ)(ε̂ − ε) equals √(ᾱ(1−ᾱ)) (z0 − x̂0). The two losses have very different magnitudes, and the entropy and orientation regularizers have fixed weights. So for the first 50 steps I scale the pixel loss to give the same gradient norm with respect to the NeRF render as latent SDS (median ratio: 10), and keep that scale for the rest of training. Everything else matches the Q2.3 corgi run: hyperparameters, seed and 10,000 iterations.

Pixel-space SDS corgi NeRF turntable: RGB and depth
"a standing corgi dog" with pixel-space SDS (left: RGB, right: depth)
Corgi: latent SDS RGB, pixel SDS RGB and pixel SDS depth over eight views
Top: latent-space SDS (Q2.3). Middle: pixel-space SDS RGB. Bottom: pixel-space SDS depth.
SDS variantOptimization time (10k iters, A100)Result
Latent space (Q2.3)2357 sSharp, saturated corgi with plausible geometry
Pixel space (L2)2672 s (+13%)Small, desaturated corgi; frontal from every view; hazy depth

Analysis