Assignment 4
Multi-View Transformers
Contents
1. Multi-View Data Preprocessing
1. Data Preprocessing
20 pointsAfter unprojection (world frame, meters)
After canonicalization
check_data.py output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 0.00010764275066321716, "centroid_distance": 1.0},
{"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
{"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
{"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00011280165199423209, "max_ray_reprojection_error_px": 0.00010225349979009479, "centroid_distance": 0.9999999403953552}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Reprojection errors are about 1e-4 px (limit 0.01) and the centroid distance is 1.
Explanations
Canonicalization. All poses are expressed relative to camera 0, so camera 0 becomes the identity. Translations, depth and points are then divided by one scene scale, the distance from camera 0 to the centroid of the valid points. This way every sample has the same reference frame and roughly the same size, and the model does not have to guess the arbitrary world origin or the absolute scale.
World-space ray directions. Each pixel direction K⁻¹[u, v, 1] is rotated by the camera rotation and normalized. Using world space instead of camera space means rays from different views live in the same frame, so the model can tell how the cameras are posed relative to each other.
Plücker moment. m = o × d. The direction alone cannot separate parallel rays from different cameras; the moment encodes where the line is (its size is the distance from the origin to the line). It is the same for any point on the ray, so [d, m] describes the line itself.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
10 points2.2 Training and Evaluation
30 pointsSettings. Default configs/lvsm.json: patch 8, width 384, 8 layers, 8 views per sample with 2 sources, batch size 4, lr 2e-4, 100 epochs, bf16. About 27 minutes on one A100.
| Source views | Conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | correct | 15.20 | 0.5586 |
| shuffled | 11.80 | 0.4875 | |
| none | 12.46 | 0.5292 | |
| 2 | correct | 16.14 | 0.5603 |
| shuffled | 11.49 | 0.4720 | |
| none | 12.44 | 0.5277 | |
| 4 | correct | 16.87 | 0.5749 |
| shuffled | 11.59 | 0.4729 | |
| none | 12.45 | 0.5378 |
Training visualizations
Evaluation visualizations
Columns: source views, ground truth, prediction.
Discussion
Source views. More sources help: PSNR goes from 15.20 (1 view) to 16.14 (2) to 16.87 dB (4), because more of the target is visible in at least one source.
Ray conditioning. Correct rays are 3–5 dB better than shuffled or zero rays, so the model really uses the target camera. With zero rays it outputs a flat average color, and its score is about the same for any number of sources. Shuffled rays are even worse than zero rays, because the model confidently renders the wrong view.
Overall quality is still low. Predictions only get the rough layout and colors, and validation PSNR was still rising at epoch 100, so the model is undertrained.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
10 points3.2 Training and Evaluation
30 pointsSettings. Default configs/vggt.json: frozen DINOv2 ViT-B/14, width 384, 8 frame/global pairs, 8 views, sampling range 32, batch size 16, lr 6e-4 with cosine decay over 1000 epochs. One shared A100. The GPU slowed down halfway through, so I stopped at epoch 722 (about 8.6 h). The results below use best.pt from epoch 721.
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.148 | 0.163 |
| Depth δ1 ↑ | 0.831 | 0.790 |
| Rotation AUC @5 / 15 / 30 | 0.178 / 0.480 / 0.671 | 0.176 / 0.461 / 0.663 |
| Translation AUC @5 / 15 / 30 | 0.009 / 0.056 / 0.156 | 0.003 / 0.034 / 0.120 |
| Joint Pose AUC @5 / 15 / 30 | 0.005 / 0.043 / 0.136 | 0.001 / 0.023 / 0.108 |
GT vs. prediction (left: GT, right: prediction)
Errors on held-out scenes
Depth is the best part: the room layout is mostly right, but surfaces are noisier and flat walls get rounded (e.g. 280b83fcf3). Translation direction is the weakest metric. In 2b1dc6d6a5 the GT cameras are spread along a path, while the predicted ones are packed into two small clusters. The 8 views come from only 32 frames, so baselines are short and hard to estimate. The camera loss was also still dropping when training stopped. Test is a bit worse than val, mainly on δ1 and translation.
3.3 Bonus: Number of Views
Bonus · 10 pointsQuestion. The model was trained with 8 views. What happens to accuracy, runtime and memory if I give it a different number of views at test time?
Setup. Same epoch-721 checkpoint, no retraining. Validation split, first 32 frames, seed 825. Only the number of views changes (2, 4, 8, 16, 24, 32). Time and memory are measured with batch size 1.
| Views | AbsRel ↓ | δ1 ↑ | Rot AUC@30 | Trans AUC@30 | Pose AUC@30 | Time (ms) | Memory (GB) |
|---|---|---|---|---|---|---|---|
| 2 | 0.117 | 0.860 | 0.644 | 0.195 | 0.162 | 48 | 0.15 |
| 4 | 0.138 | 0.834 | 0.660 | 0.183 | 0.157 | 59 | 0.17 |
| 8 | 0.148 | 0.831 | 0.671 | 0.156 | 0.136 | 72 | 0.22 |
| 16 | 0.172 | 0.798 | 0.627 | 0.137 | 0.113 | 73 | 0.30 |
| 24 | 0.175 | 0.777 | 0.638 | 0.135 | 0.113 | 75 | 0.38 |
| 32 | 0.183 | 0.771 | 0.633 | 0.133 | 0.113 | 111 | 0.47 |
Results. More views did not help. Depth and translation get steadily worse as views are added. Rotation is best at 8 views, the same number used in training. Memory grows about linearly. Runtime is flat from 8 to 24 views and jumps at 32.
Why. I think the main reason is the train/test mismatch: global attention only ever saw 8 views during training. The task also changes with the number of views. With more views, the set covers more of the 32-frame window and the scene scale changes, so the rows are not exactly the same problem.
Limitations. I only changed the number of views at test time. With 2 views there is just one pair per scene, so those pose numbers are noisy, and the timings come from a shared GPU. Training with a random number of views would be the natural next step.

























