Assignment 4: Multi-View Transformers

Name: Tashvik Dhamija (tdhamija)

1. Multi-View Data Preprocessing

1.2 Unprojection

Unprojection
Unprojected points + cameras, original world frame

1.3 Canonicalization

Canonicalization
Canonicalized points + cameras (camera 0 at origin, centroid distance ~1)

1.7 Final Check

check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

Sequence Views Resolution Valid depth fraction Max reprojection error (px) Max ray reprojection error (px) Centroid distance
e050c15a8d_frame_007080 8224 0.978 0.00011 0.00011 1.0
5f99900f09_frame_002820 8224 0.990 0.00012 0.00011 1.0
47b37eb6f9_frame_002720 8224 0.961 0.00010 0.00009 1.0
4c5c60fa76_frame_006690 8224 0.752 0.00012 0.00011 1.0

Explanation

Canonicalization: puts every sample in camera 0's frame and rescales it so geometry is consistent regardless of scene size.

World-space ray directions: each pixel's origin and unit direction expressed in the shared canonical frame, so rays from different views can be compared directly.

Plücker moment: origin cross direction, it encodes where a ray is in space without depending on which point on the ray you picked.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 31.02 dB (SSIM 0.872), after 2000 steps on one fixed batch.

Overfit visualization
val_epoch000.png (overfit)

2.2 Training and Evaluation

Settings: patch_size=8, d_model=384, n_layers=8, head_dim=64, num_source_views=2, batch_size=4, 100 epochs, lr=2e-4 with 200-step warmup + cosine decay, bfloat16 autocast.

Training curves
Train/val loss and PSNR over 100 epochs
Validation epoch 0
Validation render, epoch 0
Validation epoch 50
Validation render, epoch 50
Validation epoch 95
Validation render, epoch 95
Source viewsConditioningPSNRSSIM
1correct15.190.558
1shuffled11.810.488
1none12.460.528
2correct16.140.560
2shuffled11.490.472
2none12.440.527
4correct16.870.575
4shuffled11.590.473
4none12.450.538
Eval S1 correct
1 source view, correct conditioning
Eval S1 shuffled
1 source view, shuffled conditioning
Eval S1 none
1 source view, no conditioning
Eval S2 correct
2 source views, correct conditioning
Eval S2 shuffled
2 source views, shuffled conditioning
Eval S2 none
2 source views, no conditioning
Eval S4 correct
4 source views, correct conditioning
Eval S4 shuffled
4 source views, shuffled conditioning
Eval S4 none
4 source views, no conditioning

More source views give more context about the scene, so PSNR and SSIM go up steadily from 1 to 2 to 4 views.

Ray conditioning matters a lot. Correct conditioning gives the best results. Shuffled and no conditioning both drop sharply and perform about the same, showing the model relies on the ray geometry to know where to render, not just on the source pixels.

Visually: correct conditioning's prediction is blurry but has real local structure, varying colors and edges, consistent with an actual render attempt. None conditioning's prediction collapses into a flat, nearly textureless patch, the model has no idea where in the scene to render, so it just outputs an average blur. Shuffled conditioning's prediction also looks like a plausible blurry patch, similar in character to correct, but it's rendering the wrong location (wrong ray), so it doesn't match GT and scores almost as badly as none.

2.3 Bonus: Loss Function

Question: does adding an L1 loss term alongside the default MSE loss improve novel-view synthesis quality?

Experimental settings: same architecture, data and training config as 2.2 (patch_size=8, d_model=384, n_layers=8, batch_size=4, 100 epochs, lr=2e-4), only the loss changes: MSE-only baseline vs MSE + L1 (weight 1.0). Evaluation always uses plain MSE/PSNR/SSIM for a fair comparison, checked at both S=2 and S=4.

RunSVal loss (MSE)Val PSNRVal SSIM
Baseline (MSE only)20.0294516.140.560
+ L1 (weight 1.0)20.0288316.290.563
Baseline (MSE only)40.0247816.870.575
+ L1 (weight 1.0)40.0236717.100.580

There is a minimal but consistent increase in val PSNR and SSIM with the L1 weight added, at both S=2 and S=4.

Baseline S4 correct
Baseline (MSE only), S=4 correct
L1 S4 correct
+ L1 (weight 1.0), S=4 correct

Visually there is a bit more sharpness with L1, but it's not major enough at this scale given the experimental setup.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

Overfit loss curve
Total, camera and depth loss over 40 overfit epochs
Overfit GT
Ground truth geometry
Overfit prediction
Predicted geometry after overfitting

3.2 Training and Evaluation

python -m models.vggt.train --config configs/vggt.json --output outputs/vggt_train

Settings: 8 views, sampling range 32, batch size 16, 1000 epochs, lr 6e-4 (cosine schedule, 10-epoch warmup, min_lr_ratio 0.05), weight decay 0.01, frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (width 384, 6 heads), camera loss weight 5.0, depth loss weight 1.0.

Full training loss curve
Total, camera and depth loss over 1000 epochs
MetricValTest
Depth AbsRel0.1420.168
Depth δ10.8290.770
Rotation AUC @5°0.1770.205
Rotation AUC @15°0.4850.524
Rotation AUC @30°0.6800.718
Translation AUC @5°0.0100.015
Translation AUC @15°0.0710.070
Translation AUC @30°0.1930.169
Joint Pose AUC @5°0.0040.001
Joint Pose AUC @15°0.0540.049
Joint Pose AUC @30°0.1710.149
09c1414f1b — GT
09c1414f1b — Prediction
21d970d8de — GT
21d970d8de — Prediction
280b83fcf3 — GT
280b83fcf3 — Prediction
2b1dc6d6a5 — GT
2b1dc6d6a5 — Prediction

Depth predictions are decent, with AbsRel around 0.14 to 0.17 and about 77 to 83% of pixels within 1.25x of GT depth. Rotation is reasonably accurate at looser thresholds but weaker at the tight 5 degree threshold. Translation and joint pose AUC are the weakest numbers, since translation direction is the harder part of camera pose to get right.

Val and test metrics are close to each other, with test slightly worse on most metrics, which is expected on unseen scenes and not a sign of a big generalization gap.

Visually, predicted point clouds match the GT layout and camera positions well across all four held-out scenes. The main errors show up as extra noise and fuzziness near surface edges rather than large structural mistakes.

3.3 Bonus: Unfreezing DINO

Question: does training the DINOv2 encoder end-to-end (instead of keeping it frozen) improve Mini-VGGT's camera and depth predictions? Same architecture and loss as 3.2; only the encoder's requires_grad changes.

First checked with single-scene overfitting (40 epochs each, identical settings otherwise):

Frozen overfit loss curve
Frozen DINO (baseline)
Unfrozen overfit loss curve
Unfrozen DINO
Metric (epoch 40, overfit)FrozenUnfrozen
Loss-0.1700.087
Depth AbsRel0.0360.064
Depth δ10.9570.918
Rotation error (deg)0.180.23

Frozen DINO overfits better than unfrozen on every metric (loss -0.170 vs 0.087, depth AbsRel 0.036 vs 0.064, delta1 0.957 vs 0.918, rotation error 0.18 vs 0.23 degrees). Unfreezing means a lot more trainable parameters to fit in the same 40 epochs, so it needs a much bigger training budget to overfit as well as the frozen setup, which only has to fit the small head on top of already-good pretrained features.

Both configurations were also trained in full for 1000 epochs: the frozen baseline above (Section 3.2) and an unfrozen-encoder run with identical settings (configs/vggt_unfreeze.json).

Unfrozen full training loss curve
Unfrozen DINO, full 1000-epoch training
MetricFrozen ValUnfrozen ValFrozen TestUnfrozen Test
Depth AbsRel0.1420.2460.1680.290
Depth δ10.8290.6310.7700.562
Rotation AUC @5°0.1770.1300.2050.127
Rotation AUC @15°0.4850.3730.5240.366
Rotation AUC @30°0.6800.5680.7180.568
Translation AUC @5°0.0100.0070.0150.005
Translation AUC @15°0.0710.0380.0700.045
Translation AUC @30°0.1930.1000.1690.126
Joint Pose AUC @5°0.0040.0020.0010.001
Joint Pose AUC @15°0.0540.0270.0490.025
Joint Pose AUC @30°0.1710.0890.1490.100

Unfrozen DINO is worse than frozen across every metric on both val and test (depth AbsRel 0.246/0.290 vs 0.142/0.168, delta1 drops noticeably, rotation/translation/joint pose AUC all drop). So the gap from the overfit test holds with full training too, not just a short-run artifact.

A likely explanation is that DINOv2's pretrained features are already strong and general, while the unfrozen encoder has a lot more parameters to fit. With only 224 training scenes, that's probably not enough data to properly finetune an encoder this size, and it likely needs a lot more training budget than the frozen setup to catch up, let alone improve on it. The frozen setup is just easier to train since it only needs to fit the smaller heads on top of stable features.