Name: Tashvik Dhamija (tdhamija)


check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
| Sequence | Views | Resolution | Valid depth fraction | Max reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 0.00011 | 0.00011 | 1.0 |
| 5f99900f09_frame_002820 | 8 | 224 | 0.990 | 0.00012 | 0.00011 | 1.0 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 0.00010 | 0.00009 | 1.0 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 0.00012 | 0.00011 | 1.0 |
Canonicalization: puts every sample in camera 0's frame and rescales it so geometry is consistent regardless of scene size.
World-space ray directions: each pixel's origin and unit direction expressed in the shared canonical frame, so rays from different views can be compared directly.
Plücker moment: origin cross direction, it encodes where a ray is in space without depending on which point on the ray you picked.
Final overfit PSNR: 31.02 dB (SSIM 0.872), after 2000 steps on one fixed batch.

Settings: patch_size=8, d_model=384, n_layers=8, head_dim=64, num_source_views=2, batch_size=4, 100 epochs, lr=2e-4 with 200-step warmup + cosine decay, bfloat16 autocast.




| Source views | Conditioning | PSNR | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.558 |
| 1 | shuffled | 11.81 | 0.488 |
| 1 | none | 12.46 | 0.528 |
| 2 | correct | 16.14 | 0.560 |
| 2 | shuffled | 11.49 | 0.472 |
| 2 | none | 12.44 | 0.527 |
| 4 | correct | 16.87 | 0.575 |
| 4 | shuffled | 11.59 | 0.473 |
| 4 | none | 12.45 | 0.538 |









More source views give more context about the scene, so PSNR and SSIM go up steadily from 1 to 2 to 4 views.
Ray conditioning matters a lot. Correct conditioning gives the best results. Shuffled and no conditioning both drop sharply and perform about the same, showing the model relies on the ray geometry to know where to render, not just on the source pixels.
Visually: correct conditioning's prediction is blurry but has real local structure, varying colors and edges, consistent with an actual render attempt. None conditioning's prediction collapses into a flat, nearly textureless patch, the model has no idea where in the scene to render, so it just outputs an average blur. Shuffled conditioning's prediction also looks like a plausible blurry patch, similar in character to correct, but it's rendering the wrong location (wrong ray), so it doesn't match GT and scores almost as badly as none.
Question: does adding an L1 loss term alongside the default MSE loss improve novel-view synthesis quality?
Experimental settings: same architecture, data and training config as 2.2 (patch_size=8, d_model=384, n_layers=8, batch_size=4, 100 epochs, lr=2e-4), only the loss changes: MSE-only baseline vs MSE + L1 (weight 1.0). Evaluation always uses plain MSE/PSNR/SSIM for a fair comparison, checked at both S=2 and S=4.
| Run | S | Val loss (MSE) | Val PSNR | Val SSIM |
|---|---|---|---|---|
| Baseline (MSE only) | 2 | 0.02945 | 16.14 | 0.560 |
| + L1 (weight 1.0) | 2 | 0.02883 | 16.29 | 0.563 |
| Baseline (MSE only) | 4 | 0.02478 | 16.87 | 0.575 |
| + L1 (weight 1.0) | 4 | 0.02367 | 17.10 | 0.580 |
There is a minimal but consistent increase in val PSNR and SSIM with the L1 weight added, at both S=2 and S=4.


Visually there is a bit more sharpness with L1, but it's not major enough at this scale given the experimental setup.
python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit



python -m models.vggt.train --config configs/vggt.json --output outputs/vggt_train
Settings: 8 views, sampling range 32, batch size 16, 1000 epochs, lr 6e-4 (cosine schedule, 10-epoch warmup, min_lr_ratio 0.05), weight decay 0.01, frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (width 384, 6 heads), camera loss weight 5.0, depth loss weight 1.0.

| Metric | Val | Test |
|---|---|---|
| Depth AbsRel | 0.142 | 0.168 |
| Depth δ1 | 0.829 | 0.770 |
| Rotation AUC @5° | 0.177 | 0.205 |
| Rotation AUC @15° | 0.485 | 0.524 |
| Rotation AUC @30° | 0.680 | 0.718 |
| Translation AUC @5° | 0.010 | 0.015 |
| Translation AUC @15° | 0.071 | 0.070 |
| Translation AUC @30° | 0.193 | 0.169 |
| Joint Pose AUC @5° | 0.004 | 0.001 |
| Joint Pose AUC @15° | 0.054 | 0.049 |
| Joint Pose AUC @30° | 0.171 | 0.149 |








Depth predictions are decent, with AbsRel around 0.14 to 0.17 and about 77 to 83% of pixels within 1.25x of GT depth. Rotation is reasonably accurate at looser thresholds but weaker at the tight 5 degree threshold. Translation and joint pose AUC are the weakest numbers, since translation direction is the harder part of camera pose to get right.
Val and test metrics are close to each other, with test slightly worse on most metrics, which is expected on unseen scenes and not a sign of a big generalization gap.
Visually, predicted point clouds match the GT layout and camera positions well across all four held-out scenes. The main errors show up as extra noise and fuzziness near surface edges rather than large structural mistakes.
Question: does training the DINOv2 encoder end-to-end (instead of keeping it frozen) improve Mini-VGGT's camera and depth predictions? Same architecture and loss as 3.2; only the encoder's requires_grad changes.
First checked with single-scene overfitting (40 epochs each, identical settings otherwise):


| Metric (epoch 40, overfit) | Frozen | Unfrozen |
|---|---|---|
| Loss | -0.170 | 0.087 |
| Depth AbsRel | 0.036 | 0.064 |
| Depth δ1 | 0.957 | 0.918 |
| Rotation error (deg) | 0.18 | 0.23 |
Frozen DINO overfits better than unfrozen on every metric (loss -0.170 vs 0.087, depth AbsRel 0.036 vs 0.064, delta1 0.957 vs 0.918, rotation error 0.18 vs 0.23 degrees). Unfreezing means a lot more trainable parameters to fit in the same 40 epochs, so it needs a much bigger training budget to overfit as well as the frozen setup, which only has to fit the small head on top of already-good pretrained features.
Both configurations were also trained in full for 1000 epochs: the frozen baseline above (Section 3.2) and an unfrozen-encoder run with identical settings (configs/vggt_unfreeze.json).

| Metric | Frozen Val | Unfrozen Val | Frozen Test | Unfrozen Test |
|---|---|---|---|---|
| Depth AbsRel | 0.142 | 0.246 | 0.168 | 0.290 |
| Depth δ1 | 0.829 | 0.631 | 0.770 | 0.562 |
| Rotation AUC @5° | 0.177 | 0.130 | 0.205 | 0.127 |
| Rotation AUC @15° | 0.485 | 0.373 | 0.524 | 0.366 |
| Rotation AUC @30° | 0.680 | 0.568 | 0.718 | 0.568 |
| Translation AUC @5° | 0.010 | 0.007 | 0.015 | 0.005 |
| Translation AUC @15° | 0.071 | 0.038 | 0.070 | 0.045 |
| Translation AUC @30° | 0.193 | 0.100 | 0.169 | 0.126 |
| Joint Pose AUC @5° | 0.004 | 0.002 | 0.001 | 0.001 |
| Joint Pose AUC @15° | 0.054 | 0.027 | 0.049 | 0.025 |
| Joint Pose AUC @30° | 0.171 | 0.089 | 0.149 | 0.100 |
Unfrozen DINO is worse than frozen across every metric on both val and test (depth AbsRel 0.246/0.290 vs 0.142/0.168, delta1 drops noticeably, rotation/translation/joint pose AUC all drop). So the gap from the overfit test holds with full training too, not just a short-run artifact.
A likely explanation is that DINOv2's pretrained features are already strong and general, while the unfrozen encoder has a lot more parameters to fit. With only 224 training scenes, that's probably not enough data to properly finetune an encoder this size, and it likely needs a lot more training budget than the frozen setup to catch up, let alone improve on it. The frozen setup is just easier to train since it only needs to fit the smaller heads on top of stable features.