Assignment 4: Multi-View Transformers
1. Multi-View Data Preprocessing
1.1 Resizing camera intrinsics
s = 224 / 300 f' = s · f c' = s · (c + 0.5) − 0.5
The +0.5 is there because pixel centers are at integer coordinates, so the image edge is at −0.5.
1.2 Depth unprojection
X_world = R · ( depth(u, v) · K⁻¹ [u, v, 1]ᵀ ) + t
Points and cameras in the original world frame (meters). Red dot: camera 0. Blue dots: the other cameras.
1.3 Camera and depth canonicalization
T_i ← (T_0)⁻¹ · T_i s = ‖centroid of valid points‖ t, depth, points ← ÷ s
The same scenes after canonicalization. Each side panel shows camera 0's center, (0, 0, 0), and a valid-point centroid distance of 1.000.





1.4 Camera rays
o = t d = normalize( R · K⁻¹ [u, v, 1]ᵀ )
1.5 Plücker coordinates
plucker = [ d, m ], m = o × d
1.7 Final check
| Sample | Valid depth | Point reproj. error (px) | Ray reproj. error (px) | Centroid distance |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 97.8% | 0.000113 | 0.000109 | 1.000 |
| 5f99900f09_frame_002820 | 99.0% | 0.000123 | 0.000111 | 1.000 |
| 47b37eb6f9_frame_002720 | 96.1% | 0.000099 | 0.000089 | 1.000 |
| 4c5c60fa76_frame_006690 | 75.2% | 0.000116 | 0.000107 | 1.000 |
All reprojection errors are far below the 0.01 px limit, and the shapes match the data contract.
Full check_data.py output
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001155954014393501,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Explanations
- Canonicalization. Each sample is expressed in its first camera's frame and scaled so the valid-point centroid is 1 unit from that camera. Every sample then has the same reference frame and scale, and the model does not have to guess the arbitrary world origin or the room size.
- World-space ray directions. Each pixel's camera-space direction K−1[u, v, 1] is rotated by the camera-to-world R and normalized. Only the rotation is applied, because a direction has no position. In camera space every view would give the same map, but in the shared frame the rays show where each camera looks relative to camera 0.
- Plücker moment. m = o × d is perpendicular to the plane through the ray and the origin, and |m| is the ray's distance from the origin. Moving o along the ray does not change m, so [d, m] describes the ray itself, not the point chosen as its origin. Camera 0's moments are all zero because it sits at the origin.
2. Novel-View Synthesis with Mini-LVSM
In every visualization, the columns are: source views, ground truth, prediction.
2.1 Overfitting
After 2,000 steps on one fixed batch, the final PSNR is 30.87 dB (SSIM 0.869).

val_epoch000.png2.2 Training and evaluation
Training settings: configs/lvsm.json defaults. Patch size 8, width 384, 8 layers, head dim 64
(14.6M parameters). 2 source and 6 target views out of 8, at 224×224. Batch 4 for 100 epochs (5,600 steps).
AdamW with lr 2×10−4, weight decay 0.05, 200 warmup steps then cosine decay,
gradient clip 1.0, bf16. Training took 59 min on one NVIDIA L4. The best checkpoint is from epoch 99.

| Source views | Correct rays: PSNR / SSIM | Shuffled rays: PSNR / SSIM | No rays: PSNR / SSIM |
|---|---|---|---|
| 1 | 15.18 / 0.558 | 11.80 / 0.488 | 12.47 / 0.530 |
| 2 | 16.13 / 0.560 | 11.50 / 0.473 | 12.44 / 0.527 |
| 4 | 16.87 / 0.575 | 11.60 / 0.474 | 12.46 / 0.537 |
Validation split, best.pt.






- Number of source views. More sources help, even though the model was trained with 2: PSNR goes from 15.18 to 16.13 to 16.87 dB for 1, 2 and 4 views. Each extra view covers more of the target, so the model can copy more content instead of guessing it.
- Ray conditioning. The target rays matter. Shuffling them costs about 4.6 dB at S = 2, and removing them costs about 3.7 dB. Shuffled rays are worse than none: wrong rays make the model render a different viewpoint, while zero rays give an average, blurry image of the scene.
- Limitations. Predictions get the layout and colors right but are blurry. Validation PSNR was still rising at epoch 100, so this small model trained for an hour is undertrained.
2.3 Bonus: training view count
Question. The baseline always trains with 2 source views. Does training with a random number of source views per step make one model better at 1, 2 and 4 views?
Setup. Same config, seed and schedule as 2.2. The only change is that every batch draws S
uniformly from {1, 2, 3, 4} (configs/lvsm_random_views.json). Validation during training
uses S = 2 for both runs. Both best.pt checkpoints are evaluated with correct rays on the
validation split. The random-S run took 61 min.
| Source views at test | Fixed S = 2: PSNR / SSIM | Random S: PSNR / SSIM | Δ PSNR |
|---|---|---|---|
| 1 | 15.18 / 0.558 | 15.27 / 0.559 | +0.09 |
| 2 | 16.13 / 0.560 | 16.22 / 0.560 | +0.09 |
| 4 | 16.87 / 0.575 | 17.09 / 0.575 | +0.22 |



- Result. Random-S training is slightly better at every source count. The largest gain is +0.22 dB at S = 4, a count the baseline never trained on. Gains at S = 1 and 2 are only +0.09 dB. This is within what I'd expect from rerunning with another seed, so the two models are about equal there, and the images look the same.
- Why the gap is small. The baseline already handles 1 and 4 sources well. Source tokens get no view index, only a per-patch position, and full attention works on any sequence length. So "more source tokens" is not a new input format for the model, only more content to copy from.
- Limitations. One seed per run, 28 validation scenes. With 8 views per sample, changing S also changes the number of supervised targets per step (7 at S = 1, 4 at S = 4), so the two runs do not see exactly the same supervision.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
One fixed training clip (8 views of 036bce3393), 40 epochs × 25 steps. At the end: depth AbsRel 0.037, δ1 0.955, mean rotation error 0.22°, translation-direction error 0.43°, joint pose AUC@30 0.996.

3.2 Training and evaluation
Training settings: configs/vggt.json defaults. Frozen DINOv2 ViT-B/14, width 384,
6 heads, 8 frame/global pairs (57.7M trainable parameters). 8 views, sampling range 32, 224×224,
batch 16. 1,000 epochs (14,000 steps) with AdamW, lr 6×10−4, weight decay 0.01,
10 warmup epochs then cosine decay to 5%, gradient clip 1.0, bf16. Loss = 5 × camera + 1 × depth.
Training took 9.1 h on one NVIDIA L4. best.pt is from epoch 929 (val loss 0.872).

| Split | Depth AbsRel ↓ | Depth δ1 ↑ | Rotation AUC @5 / 15 / 30 | Translation AUC @5 / 15 / 30 | Joint pose AUC @5 / 15 / 30 |
|---|---|---|---|---|---|
| Val | 0.146 | 0.826 | 0.192 / 0.504 / 0.692 | 0.011 / 0.064 / 0.171 | 0.002 / 0.049 / 0.152 |
| Test | 0.160 | 0.801 | 0.206 / 0.525 / 0.718 | 0.014 / 0.065 / 0.158 | 0.002 / 0.042 / 0.140 |
best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825. 28 scenes and 784 view pairs per split.








- Depth generalizes best. On held-out scenes, 80–83% of pixels are within 1.25× of GT, and the predicted rooms keep their overall shape: walls, floor and furniture are in the right place.
- Rotation is moderate, with a mean pairwise error of 12.9° (val) and 9.8° (test).
- Translation direction is the main error, with a mean of 44° (val) and 43° (test). It pulls joint pose AUC@30 down to about 0.15. The 8 views come from a short handheld clip: the median distance between two val cameras is 0.29 m, and 10% of pairs are under 5 cm apart. A small position error on a short baseline is a large error in its direction. In the screenshots the predicted cameras sit in the right cluster, but their arrangement inside it is off. The misaligned views then show up as smeared, doubled surfaces, for example the floor of 2b1dc6d6a5.
- The training camera loss was still falling at epoch 1,000, so camera accuracy would likely improve with longer training.
3.3 Bonus: number of input views
Question. How does the number of input views change accuracy, runtime and memory of the model trained with 8 views?
Setup. The same best.pt on the val split with 2, 4, 8 and 16 views from the first 32
prepared frames. Batch 4, 224 px, seed 825, no retraining. The loader always takes frame 0 plus a
seeded random prefix of the other frames, so the view sets are nested (2 ⊂ 4 ⊂ 8 ⊂ 16).
That makes a controlled comparison possible: besides scoring all views, I also score only the 2
views every run shares, so only the extra context views change. Time is the model's forward pass
per sample. Memory is the peak allocated GPU memory, which includes 0.58 GB of weights (144M parameters, fp32).
| Views | All views: AbsRel / δ1 / joint AUC@30 | Shared 2 views: AbsRel / δ1 / rot. AUC@30 / trans. AUC@30 / joint AUC@30 | Time per sample (ms) | Peak memory (GB) |
|---|---|---|---|---|
| 2 | 0.130 / 0.840 / 0.254 | 0.130 / 0.840 / 0.701 / 0.277 / 0.254 | 8.3 | 0.80 |
| 4 | 0.142 / 0.835 / 0.170 | 0.134 / 0.848 / 0.706 / 0.208 / 0.198 | 13.3 | 0.88 |
| 8 (training) | 0.146 / 0.826 / 0.152 | 0.138 / 0.850 / 0.710 / 0.213 / 0.195 | 26.9 | 1.02 |
| 16 | 0.173 / 0.779 / 0.132 | 0.166 / 0.809 / 0.704 / 0.231 / 0.205 | 60.3 | 1.32 |

- Accuracy. More views do not help this model. On the shared pair, rotation stays at about 0.70–0.71. Translation AUC@30 is best with only that pair (0.277) and drops to 0.21–0.23 with extra views. Depth δ1 rises slightly at 4–8 views (0.840 → 0.850), but AbsRel rises too. At 16 views, twice the training count, depth gets clearly worse (AbsRel 0.166, δ1 0.809). With all views scored, the averages fall faster, because more views add more short-baseline pairs, which are the hardest for translation.
- Runtime and memory. Time per sample roughly doubles with each doubling of views (8 → 13 → 27 → 60 ms), and the step from 8 to 16 views is more than double. Global attention over V·N tokens grows quadratically, while the frame attention, DINO encoder and heads grow linearly. Activation memory (peak minus 0.58 GB of weights) grows from 0.22 to 0.74 GB at batch 4.
- Why. The model only saw 8-view inputs, and its weak point is translation direction on short baselines. Extra views from the same 32-frame window add more short baselines rather than wider ones, and 16 views is out of the training distribution. Training with a variable number of views, or sampling views over a wider frame range, would test whether more context can help.
- Limitations. One checkpoint and 28 validation scenes. The 2-view runs use only 28 view pairs, so their AUCs are noisy.






