16-825 Assignment 4: Multi-View Transformers
Andrew ID: tzuyiy
1. Multi-View Data Preprocessing
Visualizations


Final check
Output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4:
| Sequence | Valid depth | Max point reproj. (px) | Max ray reproj. (px) | Centroid distance |
|---|---|---|---|---|
e050c15a8d_frame_007080 | 0.978 | 1.11e-04 | 9.65e-05 | 1.0000 |
5f99900f09_frame_002820 | 0.990 | 1.11e-04 | 1.08e-04 | 1.0000 |
47b37eb6f9_frame_002720 | 0.961 | 1.07e-04 | 8.45e-05 | 1.0000 |
4c5c60fa76_frame_006690 | 0.752 | 1.14e-04 | 9.86e-05 | 1.0000 |
Both reprojection errors are around 1e-4 px (requirement: < 0.01 px) and the valid-point centroid is at distance 1 in every sample.
rgb (2, 8, 3, 224, 224) depth (2, 8, 1, 224, 224) depth_valid (2, 8, 1, 224, 224) K (2, 8, 3, 3) frame_numbers (2, 8) T_world_from_camera (2, 8, 4, 4) original_world_from_reference (2, 4, 4) scene_scale (2,) points_world (2, 8, 3, 224, 224) ray_origins (2, 8, 3, 224, 224) ray_directions (2, 8, 3, 224, 224) plucker (2, 8, 6, 224, 224)
Explanations
Canonicalization. Every pose is re-expressed relative to camera 0, so camera 0 becomes the identity. Translations, depth and points are then divided by the distance from the origin to the centroid of valid points. Every sample thus has the same reference frame and scale, regardless of the original world origin or room size.
World-space ray directions. K-1[u, v, 1] gives each pixel's direction in camera space. It is rotated by the camera-to-world R and normalized; translation is not applied because a direction has no position. All rays of a view start at that camera's center.
Plücker moment. A ray is stored as [d, m] with m = o × d. Moving o along the ray does not change m, so [d, m] describes the line itself rather than a chosen point. m is normal to the plane through the ray and the origin, and |m| is the ray's distance from the origin.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
Final overfit result after 2000 steps on one fixed batch: PSNR 30.89 dB, SSIM 0.8692.

val_epoch000.png after overfitting. Columns: source views, ground truth, prediction2.2 Training and evaluation
Settings. Default configs/lvsm.json: patch size 8, width 384, 8 layers, head dim 64, QK-norm (14.6 M parameters); 2 source views and 6 targets out of 8 views; batch size 4; 100 epochs (56 steps each); AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warm-up steps then cosine decay; gradient clipping 1.0; bf16 autocast. Trained on one NVIDIA RTX 5080 (16 GB).

Best validation PSNR 16.14 dB / SSIM 0.5597 at epoch 99 (100 epochs logged, about 24 min).

Ablations (validation split, best checkpoint)
| Source views S | Target-ray conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.5578 |
| 1 | shuffled | 11.81 | 0.4874 |
| 1 | none | 12.46 | 0.5290 |
| 2 | correct | 16.14 | 0.5597 |
| 2 | shuffled | 11.48 | 0.4716 |
| 2 | none | 12.44 | 0.5279 |
| 4 | correct | 16.87 | 0.5744 |
| 4 | shuffled | 11.59 | 0.4727 |
| 4 | none | 12.44 | 0.5386 |









Discussion
Number of source views. With correct rays, PSNR rises from 15.19 to 16.14 to 16.87 dB for S = 1, 2, 4, even though the model was trained only at S = 2. Extra views are just more source tokens, and they cover more of the target, so less has to be guessed. The gain shrinks from +1.0 dB (1 to 2 views) to +0.7 dB (2 to 4 views).
Ray conditioning. Removing or corrupting the target rays costs 2.7–5.3 dB. With zero rays the model does not know which view to render, so it outputs a nearly uniform average image and PSNR stays near 12.45 dB at every S. Shuffled rays are worse (11.5–11.8 dB): the model renders the wrong viewpoint, and misplaced content costs more under MSE than blur.
Quality. Predictions get the coarse layout and colors right (bright windows, dark furniture) but are blurry and blocky, and small objects are lost. The best validation PSNR comes in the final epoch, so this small model is likely under-trained.
2.3 Bonus: analysis and exploration
Question. The baseline always trains with S = 2. Does drawing a random number of source views per step make the model better at 1, 2 and 4 views?
Settings. Same config, seed and schedule as 2.2. The only change is that each step draws S uniformly from {1, 2, 3, 4} and uses the other 8 − S views as targets (configs/lvsm_random_s.json). Validation during training uses S = 2. Both best.pt checkpoints are evaluated with the same 3×3 ablation on the validation split.

| Eval S | Rays | Baseline PSNR / SSIM | Random S PSNR / SSIM | Δ PSNR |
|---|---|---|---|---|
| 1 | correct | 15.19 / 0.558 | 15.21 / 0.558 | +0.01 |
| 1 | shuffled | 11.81 / 0.487 | 11.78 / 0.486 | -0.03 |
| 1 | none | 12.46 / 0.529 | 12.46 / 0.531 | +0.00 |
| 2 | correct | 16.14 / 0.560 | 16.17 / 0.559 | +0.04 |
| 2 | shuffled | 11.48 / 0.472 | 11.45 / 0.469 | -0.03 |
| 2 | none | 12.44 / 0.528 | 12.43 / 0.528 | -0.00 |
| 4 | correct | 16.87 / 0.574 | 17.00 / 0.573 | +0.13 |
| 4 | shuffled | 11.59 / 0.473 | 11.50 / 0.465 | -0.09 |
| 4 | none | 12.44 / 0.539 | 12.47 / 0.537 | +0.02 |
Results. Best validation PSNR at S = 2 is 16.14 dB (baseline) vs. 16.17 dB (random S). The largest change is +0.13 dB (S = 4, correct rays); every other setting changes by less than 0.1 dB, and SSIM by less than 0.01.


The images agree with the metrics. At S = 4 both models give almost the same blurry, blocky prediction: the windows and the armchair are roughly in place, while the dining set becomes a smudge and the bookshelf is lost. Neither model is visibly sharper.
Conclusion and limitations. Random S is not meaningfully better. A possible explanation is that source tokens have no view index and attention handles any sequence length, so the S = 2 model already transfers to 1 and 4 views. Limitations: one seed per run and 28 validation scenes, so differences this small may be noise, and the images show only one scene. Changing S also changes the number of targets per step (7 at S = 1, 4 at S = 4), so the runs do not see exactly the same supervision.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting

Over 41 epochs of 25 steps on one fixed clip, the camera loss fell from 0.342 to 0.0025 and depth AbsRel from 0.619 to 0.036. The total and depth losses go negative because of the confidence term −0.2·log(c).


3.2 Training and evaluation
Settings. Default configs/vggt.json: frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads (57.7 M trainable parameters, 144.3 M in total); 8 views from a 32-frame window, 224 px; batch size 16; 1000 epochs, lr 6e-4 with 10 warm-up epochs then cosine decay to 5%, weight decay 0.01; loss weights camera 5, depth 1; bf16. Trained on one NVIDIA RTX 5080 (16 GB), about 11 s per epoch. Training was paused once after epoch 500 and resumed from last.pt; the second half used activation checkpointing, which saves memory without changing the computed values.

Evaluated with best.pt on 8 views from the first 32 prepared frames, 224 px, seed 825.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1438 | 0.1748 |
| Depth δ1 ↑ | 0.8362 | 0.7420 |
| Rotation AUC@5° ↑ | 0.1709 | 0.1814 |
| Rotation AUC@15° ↑ | 0.4769 | 0.4889 |
| Rotation AUC@30° ↑ | 0.6571 | 0.6962 |
| Translation AUC@5° ↑ | 0.0138 | 0.0107 |
| Translation AUC@15° ↑ | 0.0829 | 0.0514 |
| Translation AUC@30° ↑ | 0.1906 | 0.1381 |
| Joint pose AUC@5° ↑ | 0.0020 | 0.0013 |
| Joint pose AUC@15° ↑ | 0.0531 | 0.0293 |
| Joint pose AUC@30° ↑ | 0.1520 | 0.1165 |








Discussion
Depth generalizes best: AbsRel 0.144 / 0.175 and δ1 0.84 / 0.74 on val / test. Predictions keep the global layout (floor, walls, the corridor in scene 3) but are smoother and noisier than GT, and cluttered structures merge: the table and chairs in scene 1 become one blob of points. Scale can also drift, with a predicted centroid distance of 0.969 and 0.895 in scenes 1 and 3 instead of 1.
Rotation is moderate, with a mean error of 13.6° / 10.5° and AUC@30° of 0.66 / 0.70.
Translation direction is the main error, with a mean of about 42° / 43° and AUC@30° of 0.19 / 0.14, which pulls joint pose AUC@30° down to 0.15 / 0.12. The views come from a short handheld clip, so baselines are short, and a small position error on a short baseline is a large error in direction.
The val / test depth gap may reflect scene differences and training variance with only 28 scenes per split; test rotation is better than val, so test is not uniformly harder or easier.
3.3 Bonus: analysis and exploration
Question. How do accuracy, runtime and memory change when a model trained with V = 8 is given 2, 4, 16 or 32 views at evaluation?
Settings. The best.pt from 3.2, no retraining, on all val and test scenes, 224 px, seed 825 (scripts/vggt_view_sweep.py). Views come from the first 32 frames and are nested across V, so only the view count changes. Time is the median forward pass per scene (batch size 1, RTX 5080); activation memory is peak allocated memory beyond the model weights.

| Split | V | AbsRel ↓ | δ1 ↑ | Rot AUC@30 ↑ | Trans AUC@30 ↑ | Pose AUC@30 ↑ | ms / scene | Act. mem (GiB) |
|---|---|---|---|---|---|---|---|---|
| val | 2 | 0.125 | 0.834 | 0.676 | 0.196 | 0.138 | 9.7 | 0.14 |
| val | 4 | 0.137 | 0.838 | 0.609 | 0.174 | 0.127 | 10.1 | 0.15 |
| val | 8 | 0.144 | 0.836 | 0.657 | 0.191 | 0.152 | 13.5 | 0.18 |
| val | 16 | 0.169 | 0.790 | 0.642 | 0.163 | 0.135 | 23.6 | 0.25 |
| val | 32 | 0.173 | 0.779 | 0.639 | 0.158 | 0.131 | 48.2 | 0.40 |
| test | 2 | 0.135 | 0.815 | 0.644 | 0.112 | 0.093 | 11.8 | 0.14 |
| test | 4 | 0.155 | 0.786 | 0.688 | 0.156 | 0.120 | 9.7 | 0.15 |
| test | 8 | 0.175 | 0.742 | 0.696 | 0.138 | 0.116 | 13.7 | 0.18 |
| test | 16 | 0.181 | 0.729 | 0.708 | 0.159 | 0.136 | 23.6 | 0.25 |
| test | 32 | 0.184 | 0.741 | 0.719 | 0.162 | 0.141 | 48.3 | 0.40 |
Results. Depth gets steadily worse as V grows (AbsRel from V = 2 to 32: val 0.125 → 0.173, test 0.135 → 0.184). Pose differs by split: on test, rotation AUC@30° rises from 0.64 to 0.72 and joint pose AUC@30° from 0.09 to 0.14; on val, both stay roughly flat. Time is nearly flat up to V = 8 (about 10–14 ms), then about doubles with each doubling of V (24 and 48 ms). Activation memory grows from 0.14 to 0.40 GiB.
Explanation. A possible reason for the depth trend is that global attention mixes in more views, and the model only ever saw sequences of 8; with V = 2 the task is closer to single-image depth, which the DINOv2 features already support. More views give more overlap for registering cameras, which may explain the test pose gain.
Conclusion and limitations. The model runs at unseen view counts, but extra views do not improve depth, and any pose gain is not consistent across splits. Limitations: one checkpoint, 28 scenes per split, and only V at evaluation was changed. V = 2 has just one pose pair per scene (496 at V = 32), so small-V pose numbers are noisy. Training with a variable V would be the natural next test.