16-825 Assignment 4: Multi-View Transformers
Andrew ID: skorada
1. Multi-View Data Preprocessing
Canonicalization transforms every view into the first camera's frame and divides translations, depth, and points by one shared scene scale. Camera 0 is identity, and the valid-point centroid is one unit from the origin.
1.1 Unprojection
1.2 Canonicalization
1.3 Data validation
sequence views valid depth point reproj. ray reproj. centroid
e050c15a8d_frame_007080 8 0.9779 0.000110 0.000094 1.000000
5f99900f09_frame_002820 8 0.9903 0.000111 0.000111 1.000000
47b37eb6f9_frame_002720 8 0.9610 0.000094 0.000089 1.000000
4c5c60fa76_frame_006690 8 0.7517 0.000107 0.000094 1.000000
Batch shapes: rgb (2,8,3,224,224), depth (2,8,1,224,224), K (2,8,3,3),
poses (2,8,4,4), points/ray origins/ray directions (2,8,3,224,224),
Plücker rays (2,8,6,224,224).
1.4 Geometry explanation
World-space ray directions are obtained by applying the camera-to-world rotation
to each normalized camera ray; translation does not affect a direction. Each ray
is represented by Plücker coordinates [d, m], where d is
the unit direction and m = o × d is the moment of the ray about the
world origin.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
Final overfit PSNR: 30.84 dB.
2.2 Training and Evaluation
I trained for 1,000 epochs with 2 source views, batch size 4, 8×8 patches, width 384, 8 transformer layers, learning rate 2×10−4, weight decay 0.05, 200 warmup steps, cosine decay, and bfloat16 autocast. The best checkpoint was epoch 969 with validation PSNR 17.70 dB and SSIM 0.621.
Source-view and ray-conditioning ablations
| Source views | Conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | Correct | 16.21 | 0.6015 |
| 1 | Shuffled | 11.56 | 0.4766 |
| 1 | None | 12.44 | 0.5252 |
| 2 | Correct | 17.70 | 0.6206 |
| 2 | Shuffled | 11.15 | 0.4533 |
| 2 | None | 12.43 | 0.5234 |
| 4 | Correct | 19.03 | 0.6472 |
| 4 | Shuffled | 11.22 | 0.4471 |
| 4 | None | 12.40 | 0.5324 |









Discussion: With correct rays, increasing sources from 1 to 4 adds 2.82 dB PSNR and 0.0458 SSIM. Shuffled rays score 11.15–11.56 dB, below zeroed rays at about 12.4 dB, because they request the wrong viewpoint.
2.3 Bonus: L1-augmented reconstruction
Question: Does adding a small L1 term improve structural similarity
and sharpness without sacrificing PSNR? Both runs use the same architecture, split,
seed, optimizer, and 1,000-epoch schedule; only the objective changes from MSE to
MSE + 0.1 × L1.
| Objective | Validation PSNR ↑ | Validation SSIM ↑ |
|---|---|---|
| MSE baseline | 17.697 | 0.6206 |
| MSE + 0.1 L1 | 17.748 | 0.6262 |
Conclusion: The L1 term produced a small but consistent improvement: +0.051 dB PSNR and +0.0055 SSIM. The gain is small relative to epoch variation; repeated seeds and a perceptual metric are needed to support a sharpness claim.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
3.2 Training and Evaluation
I trained on 8 views sampled from a 32-frame window at 224×224 resolution, batch size 16, for 1,000 epochs. The model uses width 384, 8 frame/global attention pairs, 6 heads, frozen DINOv2 ViT-B/14 features, learning rate 6×10−4, weight decay 0.01, and bfloat16 precision.
Validation and test metrics
| Split | AbsRel ↓ | δ1 ↑ | Rot. AUC 5° / 15° / 30° ↑ | Trans. AUC 5° / 15° / 30° ↑ | Joint AUC 5° / 15° / 30° ↑ |
|---|---|---|---|---|---|
| Validation | 0.1566 | 0.8009 | 0.1913 / 0.5107 / 0.7002 | 0.0099 / 0.0794 / 0.1770 | 0.0046 / 0.0603 / 0.1571 |
| Test | 0.1653 | 0.7869 | 0.1796 / 0.5087 / 0.7112 | 0.0077 / 0.0536 / 0.1557 | 0.0018 / 0.0348 / 0.1418 |
Held-out geometry




Error analysis: Large surfaces and room layout are recovered, but fine geometry is smoothed and some view boundaries separate. Rotation is stronger than translation (validation AUC@30° 0.700 versus 0.177); failures concentrate around occlusions, thin or textureless structures, and limited-parallax views.
3.3 Bonus: Direct point-map prediction
Question: Can direct world-point prediction reduce the view-boundary errors from separately predicted cameras and depth? The bonus adds an RGB-refined three-channel decoder and masked, trimmed Smooth L1 point loss with weight 1. Both models use the same data, architecture, optimizer, and 1,000-epoch schedule; the bonus checkpoint is epoch 858.
| Validation model | AbsRel ↓ | δ1 ↑ | Rot. AUC 5° / 15° / 30° ↑ | Trans. AUC 5° / 15° / 30° ↑ | Joint AUC 5° / 15° / 30° ↑ | Direct-point L2 ↓ |
|---|---|---|---|---|---|---|
| Baseline | 0.1566 | 0.8009 | 0.1913 / 0.5107 / 0.7002 | 0.0099 / 0.0794 / 0.1770 | 0.0046 / 0.0603 / 0.1571 | — |
| Joint + point head | 0.1416 | 0.8234 | 0.1888 / 0.5156 / 0.6991 | 0.0122 / 0.0784 / 0.1954 | 0.0046 / 0.0563 / 0.1680 | 0.3122 |
Conclusion: joint point supervision improved depth AbsRel by 9.5%, δ1 by 0.0225, translation AUC@30° by 0.0184, and joint pose AUC@30° by 0.0110. Rotation AUC@30° stayed unchanged. The direct cloud captures room layout but smooths fine structure. One run and one point-loss weight cannot separate multi-task benefit from seed variance or establish the best weighting.