16-825 Assignment 4: Multi-View Transformers
1. Multi-View Data Preprocessing
Unprojection (Q1.1–Q1.2)

Canonicalization (Q1.3)

Final check (Q1.7)
scripts/check_data.py --split train --max-samples 4 on 224 training samples:
| Sample | Valid depth fraction | Max point reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.978 | 1.11e-04 | 9.65e-05 | 1.0000 |
| 5f99900f09_frame_002820 | 0.990 | 1.11e-04 | 1.08e-04 | 1.0000 |
| 47b37eb6f9_frame_002720 | 0.961 | 9.59e-05 | 8.22e-05 | 1.0000 |
| 4c5c60fa76_frame_006690 | 0.752 | 1.08e-04 | 9.54e-05 | 1.0000 |
Batch shapes
| Field | Shape |
|---|---|
rgb | (2, 8, 3, 224, 224) |
depth | (2, 8, 1, 224, 224) |
depth_valid | (2, 8, 1, 224, 224) |
K | (2, 8, 3, 3) |
frame_numbers | (2, 8) |
T_world_from_camera | (2, 8, 4, 4) |
original_world_from_reference | (2, 4, 4) |
scene_scale | (2,) |
points_world | (2, 8, 3, 224, 224) |
ray_origins | (2, 8, 3, 224, 224) |
ray_directions | (2, 8, 3, 224, 224) |
plucker | (2, 8, 6, 224, 224) |
Explanations
Canonicalization gives every sample the same conventions: all poses are expressed in camera 0's frame, and translations, depth and points are divided by the distance from the origin to the valid-point centroid, so camera 0 is the identity and the centroid lies at distance 1. A pixel's world-space ray direction is K-1[u, v, 1]T rotated by the camera-to-world rotation R and normalized to unit length, and its origin is the camera center. The Plücker moment m = o × d is perpendicular to the plane that contains the ray and the world origin and does not depend on which point of the ray is used, so [d, m] describes the line itself.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
One fixed training batch, 2000 steps: final PSNR 30.98 dB, SSIM 0.871 (6 min on NVIDIA A100-SXM4-40GB).

2.2 Training and evaluation
- Model
- patch 8, width 384, 8 layers, head dim 64, MLP ratio 4.0, QK-norm True (14.6M parameters)
- Data
- 8 views per sample, 2 source + 6 target, sampling range 32, 224px, batch size 4
- Optimization
- AdamW, lr 0.0002, betas (0.9, 0.95), weight decay 0.05, 200 warmup steps then cosine decay, grad clip 1.0, bf16 autocast
- Schedule
- 100 epochs × 56 steps
- Hardware
- 1× NVIDIA A100-SXM4-40GB, 1.09 h, peak memory 9.7 GB

Validation visualizations during training (columns: source views, ground truth, prediction):
Ablations: source views and target-ray conditioning (validation split)
| Source views | Correct rays | Shuffled rays | No rays | Target images | |||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | ||
| 1 | 15.19 | 0.558 | 11.81 | 0.487 | 12.46 | 0.527 | 196 |
| 2 | 16.13 | 0.560 | 11.50 | 0.472 | 12.44 | 0.526 | 168 |
| 4 | 16.89 | 0.575 | 11.59 | 0.473 | 12.46 | 0.537 | 112 |









With the correct target rays, the reconstruction quality increases as more source views are added. Once we begin shuffling or excluding rays, the image reconstruction quality decreases, as evidenced by lower PSNR and SSIM values.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
One training clip, 40 epochs × 25 steps (5 min): depth δ1 0.955, AbsRel 0.037, rotation error 0.25°, translation error 0.54°, pose AUC@30 0.993.


3.2 Training and evaluation
- Model
- frozen dinov2_vitb14, 8 frame/global pairs, width 384, 6 heads (57.7M trainable parameters)
- Data
- 8 views, sampling range 32, 224px, batch size 16
- Optimization
- AdamW, lr 0.0006, weight decay 0.01, 10 warmup epochs then cosine decay to 0.05× lr, loss weights camera 5.0 / depth 1.0, bf16
- Schedule
- 1000 epochs × 14 steps;
best.ptfrom epoch 721 (lowest validation loss) - Hardware
- 1× NVIDIA A100-SXM4-40GB, 8.15 h, peak memory 14.7 GB

Metrics (best.pt; 8 views from the first 32 prepared frames, 224px, seed 825)
| Metric | Validation (28 scenes) | Test (28 scenes) |
|---|---|---|
| Depth AbsRel ↓ | 0.1426 | 0.1613 |
| Depth δ1 ↑ | 0.8285 | 0.7915 |
| Rotation AUC@5° ↑ | 0.1765 | 0.2010 |
| Rotation AUC@15° ↑ | 0.4827 | 0.5091 |
| Rotation AUC@30° ↑ | 0.6909 | 0.7088 |
| Translation AUC@5° ↑ | 0.0097 | 0.0092 |
| Translation AUC@15° ↑ | 0.0617 | 0.0511 |
| Translation AUC@30° ↑ | 0.1705 | 0.1418 |
| Joint pose AUC@5° ↑ | 0.0038 | 0.0028 |
| Joint pose AUC@15° ↑ | 0.0497 | 0.0400 |
| Joint pose AUC@30° ↑ | 0.1561 | 0.1244 |
| Mean rotation error (°) | 12.26 | 10.03 |
| Mean translation-direction error (°) | 43.79 | 41.88 |
GT vs. predicted geometry on four scenes




Validation and test results are close so the model generalizes reasonably to unseen scenes. Translation is the main failure, as AUC@30 is only 0.171 and 0.142.


