16-825 Assignment 4 — Multi-View Transformers

Multi-view data preprocessing · novel-view synthesis (Mini-LVSM) · multi-view geometry (Mini-VGGT)

Part 1 — Multi-View Data Preprocessing 20 pts

Implemented resize_intrinsics, unproject_depth, canonicalize_cameras, camera_rays, plucker_coordinates in a4data/geometry.py.

Unprojected points, colored by true RGB, and cameras in the original world frame
Unprojection: points (true RGB) + cameras, original world frame.
Canonicalized points, colored by true RGB, camera 0 at the origin
Canonicalization: camera 0 at origin, centroid at distance 1.000.
Interactive viewer (official scripts/visualize_data.py output, canonicalization stage) — drag to rotate, scroll to zoom, click a legend entry to toggle a view, RGB/depth shown per sampled view.
SampleMax reproj. error (px)Max ray reproj. error (px)Centroid distance
10.000110.000111.00000
20.000120.000110.99999994
30.000100.000091.00000
40.000110.000100.99999994

Target: reprojection error < 0.01 px, centroid distance ≈ 1 — met with large margin, 4 training samples.

Canonicalization puts every pose/depth/point into camera 0's frame and divides by one scene scale, so scene content sits at distance ≈1 regardless of room size. Rays are unprojected through K⁻¹, rotated into world space, normalized. Plücker (d, o×d) encodes each ray as a relative, scene-independent line rather than an absolute position.

Part 2 — Novel-View Synthesis with Mini-LVSM 40 pts

Implemented patchify/unpatchify, MiniLVSM.build_tokens/forward (models/lvsm/model.py), view_synthesis_loss (models/lvsm/losses.py), and the train/loss step in models/lvsm/train.py.

2.1 Overfitting

Step 2000: train PSNR 25.43 dB, val PSNR 30.99 dB (SSIM 0.8713) — meets the ≥30 dB target.

LVSM overfit visualization: source views, ground truth, prediction
Source views · ground truth · prediction.

2.2 Training and evaluation

1000 epochs, patch 8, d_model 384, 8 layers, S=2, batch 4, MSE+SSIM+VGG loss (§2.3a). Val PSNR 17.91 dB, SSIM 0.6601.

LVSM validation visualization at epoch 995, final checkpoint
Source views · ground truth · prediction, final checkpoint.

Ablation: source-view count × ray conditioning

SConditioningPSNR (dB)SSIM
1correct16.350.6295
1shuffled11.630.4932
1none12.340.5455
2correct17.910.6601
2shuffled11.230.4707
2none12.260.5419
4correct19.420.6997
4shuffled11.280.4645
4none12.290.5525
S=2 correct conditioning
S=2, correct rays
S=2 no ray conditioning
S=2, no rays

Finding:

2.3 Bonus: loss function and model capacity

2.3a changes what the loss rewards; 2.3b changes architecture capacity (patch size or width). 2.3a's L1 variant is the best Part 2 result overall; 2.3b's capacity variants, initially negative, turned positive once given a fair initialization (below).

2.3a — loss: MSE → SSIM → VGG → GAN → L1

All in losses_bonus.py / losses_bonus_l1.py (required losses.py untouched); S=2, patch 8, d_model 384, 8 layers. Each stage resumes from the previous checkpoint.

StageLossPSNRSSIMOutcome
1MSE only (350ep)17.100.5950baseline
1+SSIM (350ep)17.190.6255positive, no tradeoff
2+VGG16 feature L1 (1000ep)17.910.6601positive, visibly sharper
3+PatchGAN adversarial (40ep)15.3–16.60.55–0.65negative, unstable
4MSE→L1 base, +SSIM+VGG kept (501ep, from stage 2)18.030.6697positive, new best
Validation PSNR and SSIM across the full 1000-epoch run
Val PSNR/SSIM, full run (dashed: VGG term added at epoch 350).

Discussion.

2.3b — model capacity: patch size vs. width

Patch 8→4 (4× more, smaller tokens) and width 384→768 (~4× the weights/block), MSE+SSIM+VGG loss, S=2.

VariantParamsPSNRSSIMOutcome
Baseline (d384, patch 8)14.6M17.910.6601reference (§2.2)
Patch 4, from scratch (100ep)14.3M14.360.5681negative
Width 768, from scratch (100ep)57.6M14.120.5640negative
Patch 4, from scratch, extended (250ep)14.3M14.850.5738still negative — not a budget issue
Width 768, from scratch, extended (250ep)57.6M14.690.5816still negative — not a budget issue
Patch 4, backbone warm-started (150ep)14.3M16.330.6035much better, still below baseline
Width 768, net2net warm-started (300ep)57.6M18.110.6730beats baseline
Width 768, from-scratch final checkpoint
Width 768, from scratch (epoch 99)
Width 768, net2net warm-started, final checkpoint
Width 768, warm-started (epoch 299)

Discussion.

Part 3 — Multi-View Geometry with Mini-VGGT 40 pts

Implemented MiniVGGT.aggregate (models/vggt/model.py), CameraHead.predict/DepthHead.forward (models/vggt/heads.py), camera_loss/depth_gradient_loss/depth_loss (models/vggt/losses.py).

3.1 Overfitting

VGGT overfit loss curves: total, camera, depth
Overfit loss curves (total, camera, depth), 40 epochs.
MetricEpoch 40
Depth AbsRel0.036
Depth δ10.956
Rotation / translation error0.21° / 0.31°
Rotation AUC @5/15/30°1.0 / 1.0 / 1.0
Pose AUC @5/15/30°0.993 / 0.998 / 0.999
VGGT overfit: GT vs predicted point cloud (true RGB) and cameras
GT vs. predicted geometry, overfit scene.
Interactive viewer (same official tool) — switch the "Geometry" dropdown between ground truth and predicted camera + depth for the overfit scene.

3.2 Training and evaluation

1000 epochs (a 200-epoch checkpoint was still improving, so training continued to completion), num_views=8, sampling_range=32, batch 16.

VGGT full 1000-epoch training curves: loss, rotation error, camera accuracy
Training loss curves (total, camera, depth), full 1000-epoch run.
MetricValTest
Depth AbsRel0.1690.183
Depth δ10.7720.753
Rotation error18.66°12.00°
Translation error48.20°47.90°
Rotation AUC @5/15/30°0.132 / 0.387 / 0.5890.151 / 0.423 / 0.647
Pose AUC @5/15/30°0.001 / 0.038 / 0.1150.001 / 0.025 / 0.103
VGGT trained model (1000 epochs): GT vs predicted geometry (true RGB) on four validation scenes
GT vs. predicted geometry, four held-out scenes.

Finding:

3.3 Bonus: eval-time view count

Question. How does the number of views at eval time (no retraining) affect accuracy, runtime, memory?

Swept V ∈ {2, 4, 8, 16, 24} (trained at V=8), validation split.

VGGT accuracy, runtime, and memory vs. eval-time view count
Accuracy / runtime / memory vs. eval-time view count.
VDepth AbsRelDepth δ1Rot. AUC@30°s/scenePeak mem (MB)
20.1290.8330.5260.831062
40.1580.7860.4641.061400
8 (trained)0.1700.7710.4781.402079
160.2100.6750.4482.323438
240.2030.6970.4543.054795

Findings.