16-825 Assignment 4: Multi-View Transformers

Andrew ID: tzuyiy

1. Multi-View Data Preprocessing

Visualizations

After unprojection, original world frame (meters), scene 036bce3393. First camera center (3.3463, 4.6765, 1.4680) m, valid-point centroid distance 6.668 m
After unprojection, original world frame (meters), scene 036bce3393. First camera center (3.3463, 4.6765, 1.4680) m, valid-point centroid distance 6.668 m
After canonicalization, same scene. <b>First camera center (0.0000, 0.0000, 0.0000)</b>, scene scale 1.867 m, <b>valid-point centroid distance 1.000</b>
After canonicalization, same scene. First camera center (0.0000, 0.0000, 0.0000), scene scale 1.867 m, valid-point centroid distance 1.000

Final check

Output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4:

SequenceValid depthMax point reproj. (px)Max ray reproj. (px)Centroid distance
e050c15a8d_frame_0070800.9781.11e-049.65e-051.0000
5f99900f09_frame_0028200.9901.11e-041.08e-041.0000
47b37eb6f9_frame_0027200.9611.07e-048.45e-051.0000
4c5c60fa76_frame_0066900.7521.14e-049.86e-051.0000

Both reprojection errors are around 1e-4 px (requirement: < 0.01 px) and the valid-point centroid is at distance 1 in every sample.

rgb                            (2, 8, 3, 224, 224)
depth                          (2, 8, 1, 224, 224)
depth_valid                    (2, 8, 1, 224, 224)
K                              (2, 8, 3, 3)
frame_numbers                  (2, 8)
T_world_from_camera            (2, 8, 4, 4)
original_world_from_reference  (2, 4, 4)
scene_scale                    (2,)
points_world                   (2, 8, 3, 224, 224)
ray_origins                    (2, 8, 3, 224, 224)
ray_directions                 (2, 8, 3, 224, 224)
plucker                        (2, 8, 6, 224, 224)

Explanations

Canonicalization. Every pose is re-expressed relative to camera 0, so camera 0 becomes the identity. Translations, depth and points are then divided by the distance from the origin to the centroid of valid points. Every sample thus has the same reference frame and scale, regardless of the original world origin or room size.

World-space ray directions. K-1[u, v, 1] gives each pixel's direction in camera space. It is rotated by the camera-to-world R and normalized; translation is not applied because a direction has no position. All rays of a view start at that camera's center.

Plücker moment. A ray is stored as [d, m] with m = o × d. Moving o along the ray does not change m, so [d, m] describes the line itself rather than a chosen point. m is normal to the plane through the ray and the origin, and |m| is the ray's distance from the origin.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit result after 2000 steps on one fixed batch: PSNR 30.89 dB, SSIM 0.8692.

<code>val_epoch000.png</code> after overfitting. Columns: source views, ground truth, prediction
val_epoch000.png after overfitting. Columns: source views, ground truth, prediction

2.2 Training and evaluation

Settings. Default configs/lvsm.json: patch size 8, width 384, 8 layers, head dim 64, QK-norm (14.6 M parameters); 2 source views and 6 targets out of 8 views; batch size 4; 100 epochs (56 steps each); AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warm-up steps then cosine decay; gradient clipping 1.0; bf16 autocast. Trained on one NVIDIA RTX 5080 (16 GB).

Mini-LVSM training curves (train vs. validation)
Mini-LVSM training curves (train vs. validation)

Best validation PSNR 16.14 dB / SSIM 0.5597 at epoch 99 (100 epochs logged, about 24 min).

Validation visualization from epoch 095. Columns: source 1, source 2, ground truth, prediction
Validation visualization from epoch 095. Columns: source 1, source 2, ground truth, prediction

Ablations (validation split, best checkpoint)

Source views STarget-ray conditioningPSNR (dB)SSIM
1correct15.190.5578
1shuffled11.810.4874
1none12.460.5290
2correct16.140.5597
2shuffled11.480.4716
2none12.440.5279
4correct16.870.5744
4shuffled11.590.4727
4none12.440.5386
S = 1, correct rays
S = 1, correct rays
S = 1, shuffled rays
S = 1, shuffled rays
S = 1, none rays
S = 1, none rays
S = 2, correct rays
S = 2, correct rays
S = 2, shuffled rays
S = 2, shuffled rays
S = 2, none rays
S = 2, none rays
S = 4, correct rays
S = 4, correct rays
S = 4, shuffled rays
S = 4, shuffled rays
S = 4, none rays
S = 4, none rays

Discussion

Number of source views. With correct rays, PSNR rises from 15.19 to 16.14 to 16.87 dB for S = 1, 2, 4, even though the model was trained only at S = 2. Extra views are just more source tokens, and they cover more of the target, so less has to be guessed. The gain shrinks from +1.0 dB (1 to 2 views) to +0.7 dB (2 to 4 views).

Ray conditioning. Removing or corrupting the target rays costs 2.7–5.3 dB. With zero rays the model does not know which view to render, so it outputs a nearly uniform average image and PSNR stays near 12.45 dB at every S. Shuffled rays are worse (11.5–11.8 dB): the model renders the wrong viewpoint, and misplaced content costs more under MSE than blur.

Quality. Predictions get the coarse layout and colors right (bright windows, dark furniture) but are blurry and blocky, and small objects are lost. The best validation PSNR comes in the final epoch, so this small model is likely under-trained.

2.3 Bonus: analysis and exploration

Question. The baseline always trains with S = 2. Does drawing a random number of source views per step make the model better at 1, 2 and 4 views?

Settings. Same config, seed and schedule as 2.2. The only change is that each step draws S uniformly from {1, 2, 3, 4} and uses the other 8 − S views as targets (configs/lvsm_random_s.json). Validation during training uses S = 2. Both best.pt checkpoints are evaluated with the same 3×3 ablation on the validation split.

Validation PSNR at S = 2 for the baseline and the random-S model
Validation PSNR at S = 2 for the baseline and the random-S model
Eval SRaysBaseline PSNR / SSIMRandom S PSNR / SSIMΔ PSNR
1correct15.19 / 0.55815.21 / 0.558+0.01
1shuffled11.81 / 0.48711.78 / 0.486-0.03
1none12.46 / 0.52912.46 / 0.531+0.00
2correct16.14 / 0.56016.17 / 0.559+0.04
2shuffled11.48 / 0.47211.45 / 0.469-0.03
2none12.44 / 0.52812.43 / 0.528-0.00
4correct16.87 / 0.57417.00 / 0.573+0.13
4shuffled11.59 / 0.47311.50 / 0.465-0.09
4none12.44 / 0.53912.47 / 0.537+0.02

Results. Best validation PSNR at S = 2 is 16.14 dB (baseline) vs. 16.17 dB (random S). The largest change is +0.13 dB (S = 4, correct rays); every other setting changes by less than 0.1 dB, and SSIM by less than 0.01.

Baseline model, S = 4, correct rays
Baseline (trained with S = 2)
Random-S model, S = 4, correct rays
Random S
The same validation scene at S = 4 with correct rays. Columns: source 1–4, ground truth, prediction

The images agree with the metrics. At S = 4 both models give almost the same blurry, blocky prediction: the windows and the armchair are roughly in place, while the dining set becomes a smudge and the bookshelf is lost. Neither model is visibly sharper.

Conclusion and limitations. Random S is not meaningfully better. A possible explanation is that source tokens have no view index and attention handles any sequence length, so the S = 2 model already transfers to 1 and 4 views. Limitations: one seed per run and 28 validation scenes, so differences this small may be noise, and the images show only one scene. Changing S also changes the number of targets per step (7 at S = 1, 4 at S = 4), so the runs do not see exactly the same supervision.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit loss curves: total, camera and depth
Overfit loss curves: total, camera and depth

Over 41 epochs of 25 steps on one fixed clip, the camera loss fell from 0.342 to 0.0025 and depth AbsRel from 0.619 to 0.036. The total and depth losses go negative because of the confidence term −0.2·log(c).

Ground truth: Overfit clip (scene 036bce3393): predicted cameras and depth match the ground truth closely
Ground truth
Prediction: Overfit clip (scene 036bce3393): predicted cameras and depth match the ground truth closely
Prediction
Overfit clip (scene 036bce3393): predicted cameras and depth match the ground truth closely

3.2 Training and evaluation

Settings. Default configs/vggt.json: frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads (57.7 M trainable parameters, 144.3 M in total); 8 views from a 32-frame window, 224 px; batch size 16; 1000 epochs, lr 6e-4 with 10 warm-up epochs then cosine decay to 5%, weight decay 0.01; loss weights camera 5, depth 1; bf16. Trained on one NVIDIA RTX 5080 (16 GB), about 11 s per epoch. Training was paused once after epoch 500 and resumed from last.pt; the second half used activation checkpointing, which saves memory without changing the computed values.

Training loss curves: total, camera and depth
Training loss curves: total, camera and depth

Evaluated with best.pt on 8 views from the first 32 prepared frames, 224 px, seed 825.

MetricValidationTest
Depth AbsRel ↓0.14380.1748
Depth δ1 ↑0.83620.7420
Rotation AUC@5° ↑0.17090.1814
Rotation AUC@15° ↑0.47690.4889
Rotation AUC@30° ↑0.65710.6962
Translation AUC@5° ↑0.01380.0107
Translation AUC@15° ↑0.08290.0514
Translation AUC@30° ↑0.19060.1381
Joint pose AUC@5° ↑0.00200.0013
Joint pose AUC@15° ↑0.05310.0293
Joint pose AUC@30° ↑0.15200.1165
Ground truth: Validation scene 1 (09c1414f1b): ground truth vs. predicted cameras and depth
Ground truth
Prediction: Validation scene 1 (09c1414f1b): ground truth vs. predicted cameras and depth
Prediction
Validation scene 1 (09c1414f1b): ground truth vs. predicted cameras and depth
Ground truth: Validation scene 2 (21d970d8de): ground truth vs. predicted cameras and depth
Ground truth
Prediction: Validation scene 2 (21d970d8de): ground truth vs. predicted cameras and depth
Prediction
Validation scene 2 (21d970d8de): ground truth vs. predicted cameras and depth
Ground truth: Validation scene 3 (280b83fcf3): ground truth vs. predicted cameras and depth
Ground truth
Prediction: Validation scene 3 (280b83fcf3): ground truth vs. predicted cameras and depth
Prediction
Validation scene 3 (280b83fcf3): ground truth vs. predicted cameras and depth
Ground truth: Validation scene 4 (2b1dc6d6a5): ground truth vs. predicted cameras and depth
Ground truth
Prediction: Validation scene 4 (2b1dc6d6a5): ground truth vs. predicted cameras and depth
Prediction
Validation scene 4 (2b1dc6d6a5): ground truth vs. predicted cameras and depth

Discussion

Depth generalizes best: AbsRel 0.144 / 0.175 and δ1 0.84 / 0.74 on val / test. Predictions keep the global layout (floor, walls, the corridor in scene 3) but are smoother and noisier than GT, and cluttered structures merge: the table and chairs in scene 1 become one blob of points. Scale can also drift, with a predicted centroid distance of 0.969 and 0.895 in scenes 1 and 3 instead of 1.

Rotation is moderate, with a mean error of 13.6° / 10.5° and AUC@30° of 0.66 / 0.70.

Translation direction is the main error, with a mean of about 42° / 43° and AUC@30° of 0.19 / 0.14, which pulls joint pose AUC@30° down to 0.15 / 0.12. The views come from a short handheld clip, so baselines are short, and a small position error on a short baseline is a large error in direction.

The val / test depth gap may reflect scene differences and training variance with only 28 scenes per split; test rotation is better than val, so test is not uniformly harder or easier.

3.3 Bonus: analysis and exploration

Question. How do accuracy, runtime and memory change when a model trained with V = 8 is given 2, 4, 16 or 32 views at evaluation?

Settings. The best.pt from 3.2, no retraining, on all val and test scenes, 224 px, seed 825 (scripts/vggt_view_sweep.py). Views come from the first 32 frames and are nested across V, so only the view count changes. Time is the median forward pass per scene (batch size 1, RTX 5080); activation memory is peak allocated memory beyond the model weights.

Accuracy and inference time vs. number of input views (val and test)
Accuracy and inference time vs. number of input views (val and test)
SplitVAbsRel ↓δ1 ↑Rot AUC@30 ↑Trans AUC@30 ↑Pose AUC@30 ↑ms / sceneAct. mem (GiB)
val20.1250.8340.6760.1960.1389.70.14
val40.1370.8380.6090.1740.12710.10.15
val80.1440.8360.6570.1910.15213.50.18
val160.1690.7900.6420.1630.13523.60.25
val320.1730.7790.6390.1580.13148.20.40
test20.1350.8150.6440.1120.09311.80.14
test40.1550.7860.6880.1560.1209.70.15
test80.1750.7420.6960.1380.11613.70.18
test160.1810.7290.7080.1590.13623.60.25
test320.1840.7410.7190.1620.14148.30.40

Results. Depth gets steadily worse as V grows (AbsRel from V = 2 to 32: val 0.125 → 0.173, test 0.135 → 0.184). Pose differs by split: on test, rotation AUC@30° rises from 0.64 to 0.72 and joint pose AUC@30° from 0.09 to 0.14; on val, both stay roughly flat. Time is nearly flat up to V = 8 (about 10–14 ms), then about doubles with each doubling of V (24 and 48 ms). Activation memory grows from 0.14 to 0.40 GiB.

Explanation. A possible reason for the depth trend is that global attention mixes in more views, and the model only ever saw sequences of 8; with V = 2 the task is closer to single-image depth, which the DINOv2 features already support. More views give more overlap for registering cameras, which may explain the test pose gain.

Conclusion and limitations. The model runs at unseen view counts, but extra views do not improve depth, and any pose gain is not consistent across splits. Limitations: one checkpoint, 28 scenes per split, and only V at evaluation was changed. V = 2 has just one pose pair per scene (496 at V = 32), so small-V pose numbers are noisy. Training with a variable V would be the natural next test.