16-825 Assignment 4 — Multi-View Transformers
Multi-view data preprocessing · novel-view synthesis (Mini-LVSM) · multi-view geometry (Mini-VGGT)
Part 1 — Multi-View Data Preprocessing
Implemented resize_intrinsics, unproject_depth, canonicalize_cameras, camera_rays, plucker_coordinates in a4data/geometry.py.
scripts/visualize_data.py output, canonicalization stage) — drag to rotate, scroll to zoom, click a legend entry to toggle a view, RGB/depth shown per sampled view.| Sample | Max reproj. error (px) | Max ray reproj. error (px) | Centroid distance |
|---|---|---|---|
| 1 | 0.00011 | 0.00011 | 1.00000 |
| 2 | 0.00012 | 0.00011 | 0.99999994 |
| 3 | 0.00010 | 0.00009 | 1.00000 |
| 4 | 0.00011 | 0.00010 | 0.99999994 |
Target: reprojection error < 0.01 px, centroid distance ≈ 1 — met with large margin, 4 training samples.
Canonicalization puts every pose/depth/point into camera 0's frame and divides by one scene scale, so scene content sits at distance ≈1 regardless of room size. Rays are unprojected through K⁻¹, rotated into world space, normalized. Plücker (d, o×d) encodes each ray as a relative, scene-independent line rather than an absolute position.
Part 2 — Novel-View Synthesis with Mini-LVSM
Implemented patchify/unpatchify, MiniLVSM.build_tokens/forward (models/lvsm/model.py), view_synthesis_loss (models/lvsm/losses.py), and the train/loss step in models/lvsm/train.py.
2.1 Overfitting
Step 2000: train PSNR 25.43 dB, val PSNR 30.99 dB (SSIM 0.8713) — meets the ≥30 dB target.
2.2 Training and evaluation
1000 epochs, patch 8, d_model 384, 8 layers, S=2, batch 4, MSE+SSIM+VGG loss (§2.3a). Val PSNR 17.91 dB, SSIM 0.6601.
Ablation: source-view count × ray conditioning
| S | Conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 16.35 | 0.6295 |
| 1 | shuffled | 11.63 | 0.4932 |
| 1 | none | 12.34 | 0.5455 |
| 2 | correct | 17.91 | 0.6601 |
| 2 | shuffled | 11.23 | 0.4707 |
| 2 | none | 12.26 | 0.5419 |
| 4 | correct | 19.42 | 0.6997 |
| 4 | shuffled | 11.28 | 0.4645 |
| 4 | none | 12.29 | 0.5525 |


Finding:
- More source views help monotonically: 16.35→17.91→19.42 dB for S=1→2→4.
- Correct ray conditioning beats shuffled/none by 5–8 dB at every S — without the right target-pose query the model can't tell which view to render, so extra source views add nothing.
2.3 Bonus: loss function and model capacity
2.3a changes what the loss rewards; 2.3b changes architecture capacity (patch size or width). 2.3a's L1 variant is the best Part 2 result overall; 2.3b's capacity variants, initially negative, turned positive once given a fair initialization (below).
2.3a — loss: MSE → SSIM → VGG → GAN → L1
All in losses_bonus.py / losses_bonus_l1.py (required losses.py untouched); S=2, patch 8, d_model 384, 8 layers. Each stage resumes from the previous checkpoint.
| Stage | Loss | PSNR | SSIM | Outcome |
|---|---|---|---|---|
| 1 | MSE only (350ep) | 17.10 | 0.5950 | baseline |
| 1 | +SSIM (350ep) | 17.19 | 0.6255 | positive, no tradeoff |
| 2 | +VGG16 feature L1 (1000ep) | 17.91 | 0.6601 | positive, visibly sharper |
| 3 | +PatchGAN adversarial (40ep) | 15.3–16.6 | 0.55–0.65 | negative, unstable |
| 4 | MSE→L1 base, +SSIM+VGG kept (501ep, from stage 2) | 18.03 | 0.6697 | positive, new best |
Discussion.
- SSIM and VGG each win cleanly, with no tradeoff against PSNR.
- GAN (a small PatchGAN discriminator) collapsed within 20–30 epochs — the discriminator dominated before the generator learned anything useful, a textbook failure mode, not a bug.
- Swapping the base term to L1 (stage 4, keeping SSIM+VGG) gave the best result overall: MSE's squared penalty makes the optimal per-pixel prediction the blurry conditional mean over plausible outputs, while L1's linear penalty makes it the sharper conditional median.
2.3b — model capacity: patch size vs. width
Patch 8→4 (4× more, smaller tokens) and width 384→768 (~4× the weights/block), MSE+SSIM+VGG loss, S=2.
| Variant | Params | PSNR | SSIM | Outcome |
|---|---|---|---|---|
| Baseline (d384, patch 8) | 14.6M | 17.91 | 0.6601 | reference (§2.2) |
| Patch 4, from scratch (100ep) | 14.3M | 14.36 | 0.5681 | negative |
| Width 768, from scratch (100ep) | 57.6M | 14.12 | 0.5640 | negative |
| Patch 4, from scratch, extended (250ep) | 14.3M | 14.85 | 0.5738 | still negative — not a budget issue |
| Width 768, from scratch, extended (250ep) | 57.6M | 14.69 | 0.5816 | still negative — not a budget issue |
| Patch 4, backbone warm-started (150ep) | 14.3M | 16.33 | 0.6035 | much better, still below baseline |
| Width 768, net2net warm-started (300ep) | 57.6M | 18.11 | 0.6730 | beats baseline |


Discussion.
- The matched-100-epoch comparison confounds "does capacity help" with "is 100 epochs from random init even enough to find out."
- Extending the from-scratch budget to 250 epochs left both variants essentially flat, ruling out a pure budget fix.
- Warm-starting instead — patch4: copy the baseline's backbone directly (126/130 tensors are shape-identical); width768: a Net2WiderNet-style function-preserving duplication, verified to reproduce the baseline exactly before training — recovers almost all of the gap, and width768 actually surpasses the baseline.
- So the capacity was real: the from-scratch comparison was only measuring whether a bigger model can re-discover the baseline's solution within a short, equal step budget.
Part 3 — Multi-View Geometry with Mini-VGGT
Implemented MiniVGGT.aggregate (models/vggt/model.py), CameraHead.predict/DepthHead.forward (models/vggt/heads.py), camera_loss/depth_gradient_loss/depth_loss (models/vggt/losses.py).
3.1 Overfitting
| Metric | Epoch 40 |
|---|---|
| Depth AbsRel | 0.036 |
| Depth δ1 | 0.956 |
| Rotation / translation error | 0.21° / 0.31° |
| Rotation AUC @5/15/30° | 1.0 / 1.0 / 1.0 |
| Pose AUC @5/15/30° | 0.993 / 0.998 / 0.999 |
3.2 Training and evaluation
1000 epochs (a 200-epoch checkpoint was still improving, so training continued to completion), num_views=8, sampling_range=32, batch 16.
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel | 0.169 | 0.183 |
| Depth δ1 | 0.772 | 0.753 |
| Rotation error | 18.66° | 12.00° |
| Translation error | 48.20° | 47.90° |
| Rotation AUC @5/15/30° | 0.132 / 0.387 / 0.589 | 0.151 / 0.423 / 0.647 |
| Pose AUC @5/15/30° | 0.001 / 0.038 / 0.115 | 0.001 / 0.025 / 0.103 |
Finding:
- Depth converges by epoch 200 (δ1 flat at ~0.77).
- Camera pose kept improving for the full 1000-epoch budget (rotation error 24.9°→18.7° val, 19.8°→12.0° test) — the 200-epoch gap was a training-budget limit, not a ceiling.
- Translation/joint pose accuracy improved more modestly and stays the weakest axis.
3.3 Bonus: eval-time view count
Question. How does the number of views at eval time (no retraining) affect accuracy, runtime, memory?
Swept V ∈ {2, 4, 8, 16, 24} (trained at V=8), validation split.
| V | Depth AbsRel | Depth δ1 | Rot. AUC@30° | s/scene | Peak mem (MB) |
|---|---|---|---|---|---|
| 2 | 0.129 | 0.833 | 0.526 | 0.83 | 1062 |
| 4 | 0.158 | 0.786 | 0.464 | 1.06 | 1400 |
| 8 (trained) | 0.170 | 0.771 | 0.478 | 1.40 | 2079 |
| 16 | 0.210 | 0.675 | 0.448 | 2.32 | 3438 |
| 24 | 0.203 | 0.697 | 0.454 | 3.05 | 4795 |
Findings.
- Accuracy peaks at V=2, below the trained V=8, and degrades further out — more views than trained dilutes each view's tokens, the opposite of Part 2's source-view trend.
- Runtime/memory scale linearly, not quadratically, with V in this range.