16-825 Assignment 4: Multi-View Transformers
1. Multi-View Data Preprocessing
1.1–1.2 Resized intrinsics and depth unprojection
Points and cameras in the original ScanNet++ world frame (meters), after Q1.1 and Q1.2. Red frustum is the first sampled camera, blue are the other 7 views. 5 training scenes, 10000 points per view.





1.3 Canonicalization
The same frames after canonicalization. Camera 0 is now the identity (red frustum at the origin, looking down +Z) and the valid-point centroid is 1 unit from the origin. The panel on the right shows the first camera center and the centroid distance.





1.7 Final check
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [{
"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 0.00010764275066321716, "centroid_distance": 1.0
}, {
"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552
}, {
"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0
}, {
"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00011280165199423209, "max_ray_reprojection_error_px": 0.00010225349979009479, "centroid_distance": 0.9999999403953552
}],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Explanations
Canonicalization. Every sample is re-expressed in the frame of its first sampled camera: Ti' = T0−1 Ti, so camera 0 becomes the identity and the other poses are relative to it. The depth of camera 0 is then unprojected in this frame and the distance from the origin to the centroid of all valid points, s = ||mean(p)||, is used as the scene scale. Translations, depth and points are all divided by s (the centroid is not subtracted, so camera 0 stays at the origin). The ScanNet++ world frame is arbitrary per scene and room sizes vary a lot, so without this the network would have to cope with arbitrary global rotations, translations and scales. After canonicalization every sample has the same reference frame and roughly the same scale.
World-space ray directions. For pixel (u, v) the camera-space direction is K−1[u, v, 1]T (pixel centers are integer coordinates, and K is the resized intrinsic matrix from Q1.1). It is rotated into the canonical world frame with the camera-to-world rotation, d = normalize(R K−1[u, v, 1]T), and the origin of every ray is the camera center t. So each pixel knows where it looks from and in which direction, in the same coordinate frame for all views.
Plücker moment. A ray is stored as six numbers [d, m] with m = o × d. The moment is the same for any point o on the line (o + λd gives (o + λd) × d = o × d), so the 6D vector describes the line itself and not a particular origin. It is perpendicular to d, and its norm is the distance from the world origin to the line. Together, d and m encode the camera position and orientation of every pixel densely, which is what LVSM uses as its camera conditioning.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
Final overfit PSNR = 30.98 dB (SSIM 0.872, MSE 0.00085). The average training PSNR over all 2000 steps is 25.41 dB, because it includes the start of training.

2.2 Training and evaluation
Training settings: Each sample has 8 views from a 32-frame window, S = 2 sources and T = 6 targets, at 224×224, batch size 4.
100 epochs × 56 steps = 5600 steps. AdamW (lr 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps then cosine decay to 10%, gradient clipping 1.0,
bfloat16 autocast with the loss in float32. One L40S, 47 minutes. Validation uses 8 views from the first 32 prepared frames of each of the 28 validation scenes.
best.pt is the last epoch (99), since validation PSNR was still going up.

Validation metrics of best.pt
| Source views | Target rays | PSNR (dB) | SSIM | Target images |
|---|---|---|---|---|
| S = 1 | correct | 15.19 | 0.5580 | 196 |
| S = 1 | shuffled | 11.80 | 0.4872 | 196 |
| S = 1 | none | 12.47 | 0.5305 | 196 |
| S = 2 | correct | 16.13 | 0.5599 | 168 |
| S = 2 | shuffled | 11.49 | 0.4719 | 168 |
| S = 2 | none | 12.45 | 0.5285 | 168 |
| S = 4 | correct | 16.88 | 0.5751 | 112 |
| S = 4 | shuffled | 11.58 | 0.4726 | 112 |
| S = 4 | none | 12.46 | 0.5385 | 112 |
Training visualizations (val_epoch*.png; columns: source 1, source 2, ground truth, prediction):



Evaluation visualizations (eval_val/S*_*.png; columns: source views, ground truth, prediction):






Quality of the baseline. After 5600 steps the model has learned the coarse layout and colours of the target view (where the window, floor, chairs and walls are), but the predictions are very blurry and every 8×8 patch is close to a flat colour. The same model reaches 31 dB when it memorizes one batch (Q2.1), so the architecture can produce detail. With 224 training scenes and a short schedule it has not yet learned to copy texture from the sources at the right place, and predicting a blurred average is the MSE-optimal answer under that uncertainty. Training and validation PSNR were both still rising at epoch 100, so the model is under-trained rather than overfitting.
Effect of the number of source views. With the true target rays, PSNR goes from 15.19 (S = 1) to 16.13 (S = 2) to 16.88 dB (S = 4), and SSIM also goes up. More sources cover more of the target view, so less of the target has to be guessed. Using S = 4 at test time also works even though the model was trained with S = 2 only. Attention works on any number of source tokens. Because the targets are the remaining 8 − S views, the target sets differ slightly between rows, but the trend is consistent.
Effect of the ray-conditioning ablation. Removing or corrupting the target rays drops PSNR by about 3.5–5 dB for every S (e.g. S = 2: 16.13 → 11.49 dB shuffled, 12.45 dB none). With none the query tokens only carry the patch position (all-zero rays never appear in training), and the model outputs an almost uniform image in the average colour of the source views, the same for every target, with no layout at all. Shuffled is the worst, slightly below none, because the model confidently renders a different view (the wrong target) instead of hedging towards the mean, and a confident wrong image costs more MSE than a blurry average. So the model uses the target Plücker rays to decide what to render, and the images change with the camera.
2.3 Bonus: number of source views during training
Question. Does the number of source views used during training change how well the model uses 1, 2 or 4 source views at test time?
Setup. Three runs with exactly the Q2.2 settings, the same seed and the same 5600 steps. Only num_source_views in the config changes, giving T = 7, 6 and 4 targets per sample. Every
best.pt (epoch 99 for all three) is evaluated with S = 1, 2, 4 and correct rays. Within a column the target images are identical, so rows can be compared directly.
| Val PSNR (dB) / SSIM | Eval S = 1 | Eval S = 2 | Eval S = 4 |
|---|---|---|---|
| Train S = 1 | 15.11 / 0.559 | 14.90 / 0.553 | 14.64 / 0.561 |
| Train S = 2 | 15.19 / 0.558 | 16.13 / 0.560 | 16.88 / 0.575 |
| Train S = 4 | 15.29 / 0.556 | 16.40 / 0.563 | 17.46 / 0.582 |

Results.
- The model trained with S = 1 does not get better with more sources. Its PSNR even goes down (15.11 → 14.90 → 14.64 dB). It never had to combine several images during training, and a longer sequence with more source tokens changes the attention statistics it was trained on.
- Training with more sources helps in every column. Train S = 4 is best for eval S = 4 (17.46 vs 16.88 dB for train S = 2) and for eval S = 2 (16.40 vs 16.13 dB), and it does not lose anything for single-view inputs (15.29 vs 15.19 vs 15.11 dB).
- Train S = 4 sees fewer target images per step (4 instead of 6) but longer sequences, so it is about 10% slower per epoch (29 vs 26 s) and uses more memory. The extra sources give more overlapping content to copy from, which seems to teach the model to attend to and fuse matching content across views. It learns this faster than when most of the target has to be guessed.
Limitations. One seed per setting, and the differences at eval S = 1 (0.2 dB) are within run-to-run noise. All models are under-trained (PSNR 15–17 dB), so the ranking could change with a longer schedule. A random S per step would probably give the most robust model, but it needs a change to the training loop, so I kept this to a config-only comparison.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
default model (frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads), one fixed training clip with 8 views, 40 epochs × 25 steps, lr 1e-3, L40S, about 5 minutes.

| Overfit clip | Depth AbsRel | Depth δ1 | Rot. error (°) | Transl. error (°) | Pose AUC@5 | Pose AUC@30 |
|---|---|---|---|---|---|---|
| Epoch 0 | 0.619 | 0.208 | 49.7 | 90.0 | 0.000 | 0.000 |
| Epoch 40 | 0.036 | 0.956 | 0.29 | 0.45 | 0.971 | 0.995 |


The predicted cameras fall on top of the GT frustums and the point cloud has the same layout as the GT. The predicted depth is a little blockier (the dense head predicts one 14×14 patch per token and the small RGB refinement only partly sharpens the edges).
3.2 Training and evaluation
Training settings: frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs
(width 384, 6 heads), camera head with 4 cross-view blocks, depth head with RGB refinement. 8 views per sample from a 32-frame window, 224×224, batch size 16,
1000 epochs × 14 steps = 14000 steps. AdamW (lr 6e-4, weight decay 0.01), 10 warmup epochs then cosine decay to 5% of the peak lr, gradient clipping 1.0,
bfloat16 autocast. Loss = 5 × camera + 1 × depth. One NVIDIA L40S, 4.05 hours of training (about 14 s per epoch including validation).
best.pt is epoch 843, the epoch with the lowest validation loss (0.804).

Metrics of best.pt (8 views from the first 32 prepared frames, 224px, seed 825, full splits, 28 scenes and 784 pose pairs each):
| Split | Depth AbsRel ↓ | Depth δ1 ↑ | Rot AUC@5 | Rot AUC@15 | Rot AUC@30 | Transl AUC@5 | Transl AUC@15 | Transl AUC@30 | Pose AUC@5 | Pose AUC@15 | Pose AUC@30 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Validation | 0.145 | 0.820 | 0.189 | 0.545 | 0.736 | 0.014 | 0.071 | 0.178 | 0.006 | 0.060 | 0.169 |
| Test | 0.149 | 0.827 | 0.235 | 0.578 | 0.762 | 0.008 | 0.066 | 0.182 | 0.002 | 0.050 | 0.169 |
Mean pairwise errors: rotation 10.7° (val) / 8.3° (test), translation direction 42.1° / 40.2°.
GT vs predicted geometry on four validation scenes (outputs/vggt_train/visualization/index.html, view 0 shown on the right panel):








Errors on held-out scenes.
- Depth is the strongest part. δ1 is about 0.82 on both splits and the overall room layout (floor, walls, the big furniture) is right. This is the task closest to what DINOv2 features already capture. The predicted depth maps are smooth: thin structures such as the chair legs and backs in scene 0 disappear into the floor and wall, and edges have a blocky 14×14 patch pattern, because each token predicts one patch and the small refinement network only partly sharpens it. Some errors are also global scale errors, since depth is in normalized units and the scale has to be inferred from the images.
- Rotation is reasonable, translation is the weak part. Rotation AUC@30 is 0.74 / 0.76, but translation AUC@30 is only 0.18 and the mean translation direction error is about 40°. Within a 32-frame window of a handheld iPhone clip the camera mostly rotates and moves only a few centimetres, so the direction of the baseline is poorly constrained by the images and small absolute errors in the camera center become large angular errors. In scene 0, for example, the predicted camera centers are bunched together near the first camera, while the GT centers are more spread out, so the model under-estimates the baselines. Joint pose AUC is limited by translation (it takes the worse of the two errors per pair).
- Generalization gap is small. Validation and test numbers are very close and the validation loss was still slowly going down at the end, so the model is capacity- and data-limited (224 training scenes, frozen encoder) rather than overfitting. Compared with the overfit run (pose AUC@30 0.995), the gap shows how hard relative pose is on unseen rooms.
3.3 Bonus: number of input views
Question. The model is trained with 8 views. How do accuracy, runtime and memory change if it is given fewer or more views at test time?
Setup. The same best.pt from Q3.2, evaluated on the full validation split with V = 2, 4, 8 and 16 views from the first 32 prepared frames
(224px, seed 825), using the provided evaluate. Runtime is the mean of 10 forward passes on one scene (batch size 1) after a warm-up, and memory is
torch.cuda.max_memory_allocated during those passes, on an L40S (script: a4_tools/vggt_num_views.py). The V = 8 row matches the official Q3.2 evaluation.
| Views | Pose pairs | AbsRel ↓ | δ1 ↑ | Rot AUC@5 | Rot AUC@30 | Transl AUC@30 | Pose AUC@30 | Forward (ms) | Peak memory (GB) |
|---|---|---|---|---|---|---|---|---|---|
| V = 2 | 28 | 0.116 | 0.871 | 0.264 | 0.770 | 0.238 | 0.233 | 33.5 | 0.74 |
| V = 4 | 168 | 0.130 | 0.851 | 0.192 | 0.725 | 0.191 | 0.177 | 33.7 | 0.76 |
| V = 8 | 784 | 0.145 | 0.820 | 0.189 | 0.736 | 0.178 | 0.170 | 34.0 | 0.80 |
| V = 16 | 3360 | 0.178 | 0.782 | 0.178 | 0.692 | 0.178 | 0.159 | 38.0 | 0.88 |

Results.
- Accuracy goes down with more views. Depth AbsRel goes from 0.116 (V = 2) to 0.178 (V = 16), δ1 from 0.871 to 0.782, and pose AUC@30 from 0.233 to 0.159. I expected more views to help, as in the full VGGT. Two effects work against it here. First, with more views from the same 32 frames, more of the selected frames are far from frame 0, and both depth (in the reference frame and the shared scale) and relative pose are harder for distant frames. With V = 2 there is only 1 pair per scene; with V = 16 there are 120 pairs, many with wide baselines. So the per-V metrics are not over the same frames or pairs. Second, the model was only trained with 8 views, so V = 16 doubles the global-attention sequence length beyond anything seen in training, which is a distribution shift for the attention.
- Runtime and memory barely change at this scale. Forward time goes from 33.5 to 38.0 ms and peak memory from 0.74 to 0.88 GB. Most of the memory is the weights (DINOv2 ViT-B and the geometry transformer), and with 261 tokens per view even 16 views is only about 4200 tokens for global attention. With batch size 1 the GPU is underused and the time is dominated by kernel launches and the per-view DINO encoding, so the quadratic cost of global attention only starts to show between V = 8 and 16. For much larger V the global attention (O((V·N)2)) and activations would start to dominate.
Limitations. The frame subset changes with V, so this mixes the effect of the number of views with the effect of which frames are chosen. A cleaner version would fix a set of target frames and only add context views around them, or train with a random number of views. Timing is for inference with batch size 1 on one GPU and does not include training memory. There is a single checkpoint and seed.