16-825 Assignment 4: Multi-View Transformers
Abhikhya Tripathy (Andrew ID: abhikhyt)
Part 1: Data
Unprojection
Depth maps from 8 sampled views unprojected into world coordinates (blue: cameras, red: first camera).
Canonicalization
Points and cameras expressed in the first camera's frame and rescaled. The side panel reports the first camera center (0, 0, 0) and the valid-point centroid distance (≈ 1).




check_data.py output
Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4. The 4 checked samples (out of 224 train samples):
| Sample | Views | Resolution | Valid depth fraction | Max point reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 1.11e-04 | 1.07e-04 | 1.000 |
| 5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.23e-04 | 1.22e-04 | 1.000 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 1.02e-04 | 8.90e-05 | 1.000 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.09e-04 | 1.03e-04 | 1.000 |
All reprojection errors are about 1e-4 px, well below the 0.01 px requirement, and every centroid distance is 1.0. Full output (a PyTorch DataLoader worker-count warning printed before the JSON is omitted):
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011141905270051211,
"max_ray_reprojection_error_px": 0.00010682854917831719,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001220703125,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00010235906665911898,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Explanations
Canonicalization. All cameras and points are moved into the first camera's frame, so camera 0 sits at the origin with no rotation. Everything is then divided by one scale factor, the distance from the origin to the average of the valid 3D points, so that this average ends up at distance 1. This way every sample uses the same coordinates and the same scale, whatever the original world frame and room size.
World-space ray directions. For each pixel, K⁻¹ · (u, v, 1) gives the direction it looks in, in camera coordinates. Rotating it by the camera's rotation turns it into world coordinates, and it is scaled to unit length. Every ray of a view starts at that camera's center.
Plücker moment. Each ray is stored as its direction d plus its moment m = o × d, where o is the ray's origin. The moment stays the same wherever you pick o along the ray, so (d, m) describes the line itself. Its length is the line's distance from the origin.
Part 2: Mini-LVSM
Q2.1 Overfitting
Final overfit validation PSNR: 30.90 dB (SSIM 0.869).
Q2.2 Training and evaluation
Settings: patch size 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm; 2 source views; batch size 4; 100 epochs; AdamW (lr 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps, grad clip 1.0, AMP; MSE loss.
Validation visualization at epoch 95 (columns: source views, ground truth, prediction):
| Source views | Correct PSNR | Correct SSIM | Shuffled PSNR | Shuffled SSIM | None PSNR | None SSIM |
|---|---|---|---|---|---|---|
| 1 | 15.20 | 0.558 | 11.81 | 0.487 | 12.46 | 0.526 |
| 2 | 16.13 | 0.560 | 11.49 | 0.472 | 12.44 | 0.526 |
| 4 | 16.86 | 0.575 | 11.59 | 0.473 | 12.45 | 0.536 |
Source views: with correct rays, PSNR rises steadily with more source views (15.2 → 16.1 → 16.9 dB), even though training used 2. More views cover more of the target, so less has to be hallucinated.
Ray conditioning: without the target rays the model cannot know which view to render. With no rays (zeros) it falls to ≈ 12.4 dB at every S and outputs a blurry, averaged image. Shuffled rays are worse still (≈ 11.5 dB): the model confidently renders the wrong viewpoint. Neither ablation benefits from more source views, which shows the gain at S = 4 comes from geometrically correct conditioning.




Q2.3 Bonus: loss function, training view count and camera encoding
Question: how do (a) extra loss terms, (b) a random number of training source views, and (c) raw camera parameters in place of Plücker rays change quality? Each run changes one thing from the Q2.2 baseline and keeps every other setting identical (100 epochs, same seed and data).
- SSIM loss: MSE + 0.2 · (1 − SSIM).
- Perceptual loss: MSE + 0.5 · VGG perceptual loss.
- Random S: 1–4 source views sampled per training step.
- Raw camera: a 16-D per-view vector (rotation 9, translation 3, normalized fx, fy, cx, cy) copied to every pixel, in place of per-pixel Plücker rays.
| Run (correct rays) | S=1 PSNR | S=1 SSIM | S=2 PSNR | S=2 SSIM | S=4 PSNR | S=4 SSIM |
|---|---|---|---|---|---|---|
| Baseline | 15.20 | 0.558 | 16.13 | 0.560 | 16.86 | 0.575 |
| + SSIM loss | 14.78 | 0.576 | 15.44 | 0.580 | 15.82 | 0.594 |
| + Perceptual loss | 13.86 | 0.529 | 14.04 | 0.527 | 14.15 | 0.535 |
| Random S ∈ [1, 4] | 15.26 | 0.559 | 16.27 | 0.560 | 17.09 | 0.573 |
| Raw camera encoding | 15.08 | 0.559 | 15.91 | 0.558 | 16.56 | 0.570 |




Conclusions.
- SSIM loss: gives the best SSIM at every S (+0.02) but costs 0.4–1.0 dB PSNR. It trades exact pixel intensities for local structure and contrast, which MSE alone undervalues.
- Perceptual loss: hurts both metrics (≈ −1.3 to −2.7 dB), and its curves plateau early. At weight 0.5 the VGG feature term dominates the gradient and pulls the model away from pixel fidelity. A lower weight, or judging it with LPIPS, would be fairer.
- Random S: matches or beats the baseline at every S and helps most at S = 4 (+0.23 dB), where the baseline never trained. Seeing varied view counts makes the model use extra views better.
- Raw camera encoding: slightly worse everywhere (−0.1 to −0.3 dB). Plücker rays give each patch its own geometric direction, whereas the raw vector is identical for every patch, so the network must work out the per-pixel ray geometry from position embeddings itself.
Limitations: one seed per run, so differences under ≈ 0.2 dB may be noise. Only PSNR/SSIM were measured (no LPIPS), and only on the validation split.
Part 3: Mini-VGGT
Q3.1 Overfitting
After 40 overfit epochs on one scene: total loss −0.150, camera loss 0.0025, depth loss −0.163 (the depth loss is confidence-weighted, so it can go negative).


Q3.2 Training and evaluation
Settings: frozen DINOv2 ViT-B/14 encoder, 224 px; aggregator width 384, 6 heads, 8 layers; 8 views sampled within 32 frames; batch size 16; 1000 epochs; lr 6e-4 with 10 warmup epochs and cosine decay to 0.05×; weight decay 0.01; loss = 5 · camera + 1 · depth. Evaluation: 8 views from the first 32 prepared frames, 224 px, seed 825.
| Split | AbsRel ↓ | δ1 ↑ | Rot AUC 5/15/30 ↑ | Trans AUC 5/15/30 ↑ | Pose AUC 5/15/30 ↑ |
|---|---|---|---|---|---|
| Val | 0.154 | 0.792 | 0.189 / 0.485 / 0.671 | 0.016 / 0.069 / 0.179 | 0.004 / 0.049 / 0.155 |
| Test | 0.161 | 0.790 | 0.219 / 0.549 / 0.736 | 0.014 / 0.076 / 0.191 | 0.005 / 0.057 / 0.174 |
Held-out errors: depth holds up well: about 79% of pixels are within 25% of the true depth. The overall room layout matches GT, but surfaces are noisier and edges are blurred. For cameras, rotation is fairly good (mean error ≈ 13°), but translation direction is poor (mean error ≈ 43°, AUC@30 < 0.2), and it drags down the joint pose AUC. The views come from a short stretch of video, so the cameras are close together and even small position errors give large direction errors. Val and test scores are similar, so the model is not overfitting to particular scenes.








Q3.3 Bonus: multi-task synergy and number of views
(a) Multi-task synergy. Does learning cameras and depth jointly help either task? Two single-task models were trained with the Q3.2 settings, changing only the loss weights: camera-only (camera 5, depth 0) and depth-only (camera 0, depth 1). All three models used the same evaluation protocol. Grey cells are heads that received no training signal.
| Model / split | AbsRel ↓ | δ1 ↑ | Rot AUC 5/15/30 ↑ | Trans AUC 5/15/30 ↑ | Pose AUC 5/15/30 ↑ |
|---|---|---|---|---|---|
| Joint — val | 0.154 | 0.792 | 0.189 / 0.485 / 0.671 | 0.016 / 0.069 / 0.179 | 0.004 / 0.049 / 0.155 |
| Camera-only — val | 0.645 | 0.269 | 0.215 / 0.532 / 0.708 | 0.009 / 0.079 / 0.195 | 0.003 / 0.065 / 0.175 |
| Depth-only — val | 0.149 | 0.804 | 0.049 / 0.147 / 0.286 | 0 / 0 / 0 | 0 / 0 / 0 |
| Joint — test | 0.161 | 0.790 | 0.219 / 0.549 / 0.736 | 0.014 / 0.076 / 0.191 | 0.005 / 0.057 / 0.174 |
| Camera-only — test | 0.653 | 0.269 | 0.243 / 0.575 / 0.757 | 0.021 / 0.075 / 0.199 | 0.004 / 0.050 / 0.182 |
| Depth-only — test | 0.170 | 0.770 | 0.074 / 0.171 / 0.326 | 0 / 0 / 0 | 0 / 0 / 0 |


Conclusions: joint training shows no clear synergy at this scale.
- Cameras: the camera-only model is slightly better on rotation (AUC@15 0.532 vs 0.485 val; 0.575 vs 0.549 test) and on pose AUC@30, so sharing capacity with depth slightly hurts cameras.
- Depth: results are mixed. Depth-only is a little better on val (AbsRel 0.149 vs 0.154) but worse on test (0.170 vs 0.161), so the camera task may regularize depth slightly.
A likely reason is that the frozen DINO features already carry most of the geometry, and the small 8-layer aggregator is capacity-limited, so the tasks compete rather than help each other.
Limitations: single seed; differences are small relative to likely run-to-run variance; only one set of loss weights was tried.
(b) Number of views at evaluation. The Q3.2 model (trained with 8 views) was evaluated with 2, 4, 8, 16 and 32 input views on both splits, also measuring forward time and peak memory.
| Views | Val AbsRel ↓ | Val δ1 ↑ | Val Rot AUC@15 | Val Pose AUC@30 | Test AbsRel ↓ | Test Rot AUC@15 | Test Pose AUC@30 | Forward (ms) | Peak mem (GB) |
|---|---|---|---|---|---|---|---|---|---|
| 2 | 0.112 | 0.867 | 0.540 | 0.135 | 0.119 | 0.626 | 0.124 | 33.0 | 0.74 |
| 4 | 0.137 | 0.837 | 0.508 | 0.145 | 0.147 | 0.535 | 0.171 | 32.5 | 0.76 |
| 8 | 0.154 | 0.792 | 0.485 | 0.155 | 0.161 | 0.550 | 0.175 | 32.6 | 0.81 |
| 16 | 0.192 | 0.734 | 0.448 | 0.132 | 0.171 | 0.555 | 0.181 | 34.0 | 0.89 |
| 32 | 0.198 | 0.721 | 0.452 | 0.132 | 0.177 | 0.568 | 0.176 | 40.0 | 1.05 |
Conclusions:
- Depth: gets steadily worse with more views (val AbsRel 0.112 → 0.198). With more views spread over the 32-frame window, the canonical frame and scale are set by a larger, more varied scene, and the model never saw that view count in training.
- Pose: best near the training count of 8 on val, and roughly flat on test.
- Cost: grows sub-linearly. Forward time only rises from 33 to 40 ms and peak memory from 0.74 to 1.05 GB, so at these sizes the GPU is not saturated and fixed overhead dominates.
Training with a variable view count would likely be needed to benefit from more views.
Limitations: evaluation-only (no retraining at other view counts). With 2 views there is only one pose pair per sample, so those AUCs are noisy. Metrics at different view counts cover different sets of pixels and pairs.