Assignment 4: Multi-View Transformers

Krisha Chemburkar
Andrew ID: kchembur

1 Multi-view data preprocessing

Data check (scripts/check_data.py)

All checks pass on 4 training samples (224 samples in the split). Point and ray reprojection errors are about 10−4 px, well under the 0.01 px threshold. After canonicalization the valid-point centroid sits at distance 1.000 from the origin.

SequenceViewsRes.Valid depthMax point reproj. (px)Max ray reproj. (px)Centroid dist.
e050c15a8d_frame_00708082240.9781.13e-041.09e-041.000
5f99900f09_frame_00282082240.9901.23e-041.22e-041.000
47b37eb6f9_frame_00272082240.9619.78e-058.90e-051.000
4c5c60fa76_frame_00669082240.7521.17e-041.12e-041.000
Batch shapes
TensorShape
rgb(2, 8, 3, 224, 224)
depth(2, 8, 1, 224, 224)
depth_valid(2, 8, 1, 224, 224)
K(2, 8, 3, 3)
frame_numbers(2, 8)
T_world_from_camera(2, 8, 4, 4)
original_world_from_reference(2, 4, 4)
scene_scale(2)
points_world(2, 8, 3, 224, 224)
ray_origins(2, 8, 3, 224, 224)
ray_directions(2, 8, 3, 224, 224)
plucker(2, 8, 6, 224, 224)

Unprojection

Unprojection, scene 0
Unprojection, scene 0
Unprojection, scene 1
Unprojection, scene 1
Unprojection, scene 2
Unprojection, scene 2
Unprojection, scene 3
Unprojection, scene 3

Canonicalization

The first camera sits at the origin looking down +z, and the scene is in normalized units.

Canonicalization, scene 0: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 0: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 1: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 1: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 2: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 2: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 3: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000
Canonicalization, scene 3: the side panel reads the first camera center (0, 0, 0) and valid-point centroid distance 1.000

Explanations

Canonicalization
Each ScanNet++ scene has its own arbitrary world origin and scale, so absolute poses don't mean much to a network. We move everything into the first camera's frame with T′ᵢ = T₀⁻¹ Tᵢ, which puts camera 0 at the origin with identity rotation. Then we divide all translations, depths and points by s = ‖mean(valid points)‖, so the valid-point centroid ends up at distance 1. Every sample now shares the same origin, orientation and scale.
World-space ray directions
For pixel (u, v), the direction in camera coordinates is K⁻¹[u, v, 1]ᵀ, using the intrinsics after cropping and resizing. Rotating it into the world with the camera-to-world rotation R and normalizing gives d, and the ray starts at the camera center o = t. Since the world frame is the canonical first-camera frame, rays from every view share one coordinate system, which is what lets the network relate pixels across views.
Plücker moment
Each ray becomes a 6-vector (d, m), where the moment is m = o × d. Any point on the ray gives the same moment, since (o + λd) × d = o × d, so (d, m) describes the line itself, not a particular point on it. The size of m is the line's distance from the origin. Stacked per pixel, this gives the network a 6-channel image that encodes the camera's intrinsics and pose.

2 Mini-LVSM novel-view synthesis

2.1 Overfitting one batch

After 2000 steps on one fixed batch, the final overfit PSNR is 30.96 dB, above the ~30 dB target.

Overfit batch, val_epoch000.png. Columns: source views, ground truth, prediction.
Overfit batch, val_epoch000.png. Columns: source views, ground truth, prediction.

2.2 Training and evaluation

Settings. Default configs/lvsm.json, unchanged: 14.6 M parameters, patch size 8, 224×224 inputs, 8 views per sample (2 source, 6 target), batch size 4, AdamW at lr 2·10⁻⁴, 100 epochs. The best validation PSNR (16.15 dB, SSIM 0.560) came at the last epoch, so the model was still improving.

Training curves: MSE loss, PSNR and validation SSIM per epoch.
Training curves: MSE loss, PSNR and validation SSIM per epoch.
Training-time visualization, epoch 0
Training-time visualization, epoch 0
Training-time visualization, epoch 95
Training-time visualization, epoch 95

Ablation over source views and ray conditioning (validation)

Source viewsTarget raysPSNR (dB) ↑SSIM ↑Images
S=1correct15.190.5583196
S=1shuffled11.800.4877196
S=1none12.470.5292196
S=2correct16.150.5603168
S=2shuffled11.490.4723168
S=2none12.440.5266168
S=4correct16.890.5754112
S=4shuffled11.590.4730112
S=4none12.450.5364112

Each visualization row is one validation sample. Columns: the S source views, ground truth, then the prediction.

S=1, correct target rays
S=1, correct target rays
S=1, shuffled target rays
S=1, shuffled target rays
S=1, none target rays
S=1, none target rays
S=2, correct target rays
S=2, correct target rays
S=2, shuffled target rays
S=2, shuffled target rays
S=2, none target rays
S=2, none target rays
S=4, correct target rays
S=4, correct target rays
S=4, shuffled target rays
S=4, shuffled target rays
S=4, none target rays
S=4, none target rays

Discussion

2.3 Bonus: training with a random number of source views

Question. The baseline always trains with S=2 source views and is evaluated with 1, 2 and 4. Does drawing S uniformly from {1, 2, 3, 4} for each training batch (T = 8 − S targets) make the model more robust to the number of source views at test time?

Setting. Same model, data, optimizer, schedule, seed and 100-epoch budget as Q2.2 (configs/lvsm_randS.json). The only change is the per-batch random S. The checkpoint is selected by validation PSNR at S=2, as for the baseline, and both use best.pt with correct rays.

Results. Random-S training matches the baseline at every evaluation S. PSNR is +0.00 dB at S=1, +0.02 dB at S=2 and +0.10 dB at S=4, and every SSIM change is under 0.002. The validation curves at S=2 overlap for the whole run. Both models gain about 1.7 dB going from 1 to 4 source views, and the predictions look the same (blurry, patchy colour layouts).

Conclusions and explanation. Training with a random number of source views did not really help. (1) The baseline already handles different S well. The transformer just sees a set of source tokens, each tagged with its own rays, so it doesn't care much how many there are, and both models get better with more views. (2) The only small gain is at S=4 (+0.10 dB), which makes sense because the random-S model saw 3 and 4 views during training. At S=1 nothing changes, because the problem there is that one view simply doesn't show enough of the scene. (3) The random-S model also sees slightly fewer target views per step. At this size, the model's capacity and the short 100-epoch training matter more than which S it trained with.

Limitations. One seed per configuration, so differences of about 0.1 dB are within run-to-run noise. Checkpoints were selected at S=2, which favours the baseline's training setting. Only 100 epochs were run, and the gap could change for a longer-trained model that has learned to rely more on cross-view correspondence.

Eval SPSNR basePSNR rand-SΔSSIM baseSSIM rand-SΔ
S=115.1915.19+0.000.55830.5588+0.0004
S=216.1516.16+0.020.56030.5597-0.0006
S=416.8916.99+0.100.57540.5738-0.0016
LVSM random-S bonus plots
Left and middle: validation PSNR and SSIM by evaluation S. Right: validation PSNR at S=2 during training.
Baseline, S=1
Baseline, S=1
Random S, S=1
Random S, S=1
Baseline, S=4
Baseline, S=4
Random S, S=4
Random S, S=4

3 Mini-VGGT feed-forward geometry

3.1 Overfitting one scene

Overfit loss curves: total, camera and depth loss.
Overfit loss curves: total, camera and depth loss.
Overfit scene: ground truth
Overfit scene: ground truth
Overfit scene: prediction
Overfit scene: prediction

3.2 Training and evaluation

Settings. Default configs/vggt.json, unchanged: frozen DINOv2 ViT-B/14 encoder with 57.7 M trainable parameters, 8 views per sample from a 32-frame window, 224×224, batch size 16, AdamW at lr 6·10⁻⁴, 1000 epochs, loss = 5·camera + depth.

Normal training: total, camera and depth loss (train and validation) per epoch.
Normal training: total, camera and depth loss (train and validation) per epoch.

Validation and test metrics

Metrics from best.pt on the 28 validation scenes and the 28 held-out test scenes (separate splits, neither used for training).

MetricValidationTest
Depth AbsRel0.14370.1547
Depth δ10.83210.8021
Rotation error (°)12.569.29
Translation-dir. error (°)42.3441.06
Rotation AUC@50.19310.2041
Rotation AUC@150.52360.5386
Rotation AUC@300.70450.7306
Translation AUC@50.00990.0084
Translation AUC@150.07540.0504
Translation AUC@300.18850.1593
Pose AUC@50.00430.0023
Pose AUC@150.05590.0344
Pose AUC@300.16220.1451

GT vs. prediction on four held-out validation scenes

Each pair shows the same viewpoint. Left: GT cameras + GT depth unprojected. Right: predicted cameras + predicted depth, in the same canonical frame and colours.

Validation scene 0
Scene 0: ground truth
Scene 0: ground truth
Scene 0: predicted cameras and depth
Scene 0: predicted cameras and depth
Validation scene 1
Scene 1: ground truth
Scene 1: ground truth
Scene 1: predicted cameras and depth
Scene 1: predicted cameras and depth
Validation scene 2
Scene 2: ground truth
Scene 2: ground truth
Scene 2: predicted cameras and depth
Scene 2: predicted cameras and depth
Validation scene 3
Scene 3: ground truth
Scene 3: ground truth
Scene 3: predicted cameras and depth
Scene 3: predicted cameras and depth

Discussion of held-out errors

3.3 Bonus: how many input views?

Question. The model was trained with exactly V=8 views. At test time, how do accuracy, runtime and memory change when it receives 2 to 32 views of the same scene window?

Setting. One trained checkpoint (outputs/vggt_train/best.pt), validation split, the same 32-frame prepared windows (seed 825), and V ∈ {2, 4, 8, 16, 24, 32} views sampled from each window. Batch size 1. Runtime is the model forward pass (encoder + aggregator + heads, bf16), averaged over 27 samples after one warm-up. Memory is the peak allocated GPU memory. Pairwise AUCs are computed over all V(V−1) pairs, so a larger V also means more (and more widely spaced) pairs. Script: tools_report/vggt_views.py.

VAbsRel ↓δ1 ↑Rot AUC@30 ↑Trans AUC@30 ↑Pose AUC@30 ↑ms / samplePeak mem (GB)
20.11320.87020.7500.2010.18129.60.98
40.13150.85080.6970.1920.16730.70.99
80.14370.83210.7040.1890.16238.61.03
160.16790.79830.6680.1680.14266.01.08
240.16970.77630.6680.1680.143104.71.37
320.17370.76940.6590.1640.139148.81.43
VGGT view-count sweep
Accuracy, runtime and memory against the number of input views. The dashed line marks the training view count (V=8).

Results. More input views make every accuracy metric slightly worse. Depth AbsRel goes from 0.113 (V=2) to 0.144 (V=8) to 0.174 (V=32), and δ1 from 0.870 to 0.769. Rotation AUC@30 drops from 0.75 to 0.66, while translation AUC@30 stays flat at about 0.16–0.20, so pose AUC@30 goes from 0.18 to 0.14. Runtime is nearly constant up to V=8 (30–39 ms) and then grows faster than linearly (66 ms at 16, 149 ms at 32). Peak memory only rises from 0.98 to 1.43 GB, because at batch size 1 under no_grad most of it is the 144 M-parameter weights.

Explanations. (1) Train/test mismatch: the model only saw V=8. With V=32 the global-attention layers attend over 4× more tokens (32 × 256), which changes the attention statistics the camera and depth tokens were trained on. The decline is steady up to V=16 and then levels off. (2) Harder views: with V=2, one of the two views is the reference frame, which has the identity pose and is the easiest to predict depth for. Larger V adds more frames far from the reference within the 32-frame window, whose poses and scale must be inferred through the chain of views. So the metrics are not computed on identical image sets, and part of the trend comes from the views, not the model. (3) Cost: frame attention and the DINO encoder scale linearly in V, global attention quadratically in V·N. At small V, fixed overheads (kernel launches, heads) dominate, so the curve is flat. Beyond 8 views the quadratic global attention starts to show. Memory stays low thanks to the small model and inference without gradients. Training at V=32 with batch 16 would be a different story: activations would grow about 4× and would not fit in 16 GB.

Conclusions and limitations. Giving this small model more views at test time doesn't make it more accurate. It was only trained with 8 views, so it hasn't learned to use the extra views the way the full VGGT does, which trains with varying view counts.