1. Multi-View Data Preprocessing 20 pts

Implement resize_intrinsics, unproject_depth, canonicalize_cameras, camera_rays and plucker_coordinates in a4data/geometry.py. Include screenshots of the data visualizations after unprojection and canonicalization (showing the first camera center and the valid-point centroid distance), the check_data.py output, and briefly explain canonicalization, world-space ray directions and the Plücker moment.

Unprojection (original world frame, meters)

Each pixel (u, v) is lifted with X_cam = z · K-1 [u, v, 1]T (depth is optical-axis z-depth, so no ray normalization), then mapped to the world with the camera-to-world pose X_world = R X_cam + t. The eight frames of a clip land on one consistent point cloud.

Canonicalization (camera-0 frame, normalized units)

The same scenes after canonicalization. The first camera (red) sits at the origin, about (0, 0, 0), and the valid-point centroid is 1.000 unit away (side panel of each screenshot).

check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
SequenceViewsRes.Valid depthMax point reproj. err. (px)Max ray reproj. err. (px)Centroid dist.
e050c15a8d_frame_00708082240.9781.07e-49.31e-51.0
5f99900f09_frame_00282082240.9901.07e-49.65e-51.0
47b37eb6f9_frame_00272082240.9611.10e-41.08e-41.0
4c5c60fa76_frame_00669082240.7528.90e-58.91e-51.0

Batch shapes (batch of 2): rgb, points_world, ray_origins, ray_directions are (2, 8, 3, 224, 224); depth is (2, 8, 1, 224, 224); plucker is (2, 8, 6, 224, 224); original_world_from_reference is (2, 4, 4). All reprojection errors are about 1e-4 px, far below the 0.01 px threshold.

Explanations

2.1 Mini-LVSM Overfitting 10 pts

Overfit one fixed batch (--overfit 2000). Include the final overfit PSNR and the val_epoch000.png visualization.

After 2000 steps on one batch, the final overfit PSNR is 30.96 dB (SSIM 0.871, MSE 8.6e-4), above the ~30 dB target.

Overfit visualization: source views, ground truth and nearly identical predictions
Figure 5. val_epoch000.png after overfitting. Columns: source views, ground truth, prediction.

2.2 Mini-LVSM Training and Evaluation 30 pts

Train on the full training split and evaluate best.pt with 1, 2 and 4 source views and correct / shuffled / no target-ray conditioning. Include training curves, PSNR/SSIM for each setting, training and evaluation visualizations, the training settings, and discuss the effect of the number of source views and of the ray conditioning.

Training settings

Mini-LVSM (14.6M parameters): patch size 8, width 384, 8 transformer blocks, head dimension 64, MLP ratio 4, QK-norm. AdamW (lr 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps then cosine decay to 10%, gradient clipping 1.0, bf16 autocast. Batch size 4, 8 views per clip at 224×224 (2 source + 6 target views), sampling range 32. I trained the default 100 epochs first (53 min), and since validation PSNR was still rising, a second run for 300 epochs (56 steps/epoch, 2.6 h on a 16 GB laptop GPU). Results below use the 300-epoch run (best validation PSNR at epoch 297).

300-epoch training curves: MSE loss falls and validation PSNR rises to 16.94 dB
Figure 6. Train/validation MSE loss and PSNR, 300-epoch run.
100-epoch training curves, validation PSNR reaching about 16.1 dB
Figure 7. The default 100-epoch run for reference.

Ablations (validation, 300-epoch best.pt)

python -m models.lvsm.eval --checkpoint outputs/lvsm_long/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none
Source viewsCorrect rays
PSNR / SSIM
Shuffled rays
PSNR / SSIM
No rays
PSNR / SSIM
115.77 / 0.58211.66 / 0.48312.48 / 0.534
216.94 / 0.59111.26 / 0.46412.42 / 0.530
418.07 / 0.61011.36 / 0.46312.43 / 0.540

For comparison, the 100-epoch model with correct rays scores 15.20 / 16.14 / 16.88 dB for 1 / 2 / 4 source views.

Training visualizations

Evaluation visualizations

Columns: source views, ground truth, prediction.

Discussion

2.3 Bonus: Mini-LVSM Exploration 10 pts

Describe your question, experimental settings, results and conclusions, with plots or screenshots, possible explanations and limitations.

Question: does training with a random number of source views help?

The Q2.2 model is always trained with S = 2 sources, yet it is evaluated with 1, 2 and 4. I tested whether exposing the model to every evaluation setting during training, by drawing S uniformly from {1, 2, 3, 4} at each step (the remaining 8 − S views are targets), improves the 1- and 4-view results without hurting 2 views.

Setup

A controlled comparison against the default 100-epoch S = 2 run: identical model, optimizer, schedule, data, seed and epoch count (100 epochs, 56 steps/epoch); only the per-step S differs. Validation during training still uses S = 2, and both best.pt checkpoints are evaluated with the same Q2.2 command. Change: about five lines in train_one_epoch, enabled by "random_source_views": [1, 4] in configs/lvsm_randS.json. Cost: 33 s/epoch vs 32 s/epoch.

python -m models.lvsm.train --config configs/lvsm_randS.json
python -m models.lvsm.eval --checkpoint outputs/lvsm_randS/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none

Results (validation, correct rays)

Source views at evalFixed S = 2
PSNR / SSIM
Random S ∈ {1..4}
PSNR / SSIM
Δ PSNR
115.20 / 0.55815.28 / 0.560+0.08
216.14 / 0.56016.24 / 0.560+0.10
416.88 / 0.57517.08 / 0.575+0.20

The ray ablations behave as before for the random-S model: shuffled rays give 11.4–11.8 dB and no rays about 12.5 dB at every S.

Training curves of the random-S model, similar in shape to the fixed-S run
Figure 23. Train/validation curves with random S (train loss mixes S = 1–4; val uses S = 2). Compare with Figure 7.

Conclusions and limitations

3.1 Mini-VGGT Overfitting 10 pts

Overfit one fixed training clip. Include the overfit loss curve (total, camera and depth loss) and screenshots comparing GT and predicted geometry.

After 40 epochs × 25 steps on one clip, the camera loss falls to 0.0025 and the depth loss goes negative (the confidence term −0.2 log c lets it drop below zero). On the fitted clip: rotation error 0.22°, translation error 0.42°, pose AUC@30 0.995, depth AbsRel 0.036, δ1 0.957. The predicted geometry closely matches GT.

Overfit loss curves: total, camera and depth loss all decreasing
Figure 11. Overfit loss curves.

3.2 Mini-VGGT Training and Evaluation 30 pts

Train on the full training split and evaluate best.pt on validation and test. Include the loss curve, validation/test metrics, GT/prediction screenshots for four different scenes, the training settings, and discuss errors on held-out scenes.

Training settings

Default configs/vggt.json: frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (width 384, 6 heads), camera head with 4 cross-view blocks, depth head with RGB refinement. Loss weights camera 5.0, depth 1.0. AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs then cosine decay to 5%, bf16. Batch size 16, 8 views, sampling range 32, 224×224, seed 825, 1000 epochs (14 steps/epoch). Training took 6.5 h on a single 16 GB laptop GPU. best.pt (lowest validation loss) is from epoch 929.

Training curves: total, camera and depth loss decreasing over 1000 epochs
Figure 14. Train/validation total, camera and depth loss.

Metrics

python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split {val,test} --eval-root data/scannetpp/prepared_eval \
    --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825
MetricValidation (28 scenes)Test (28 scenes)
Total / camera / depth loss0.789 / 0.112 / 0.2310.759 / 0.100 / 0.260
Depth AbsRel ↓0.1270.151
Depth RMSE (normalized) ↓0.1760.193
Depth δ1 ↑0.8610.821
Mean rotation error ↓12.24°10.92°
Mean translation-direction error ↓42.43°42.02°
Rotation AUC @5 / 15 / 30 ↑0.187 / 0.505 / 0.6960.180 / 0.473 / 0.674
Translation AUC @5 / 15 / 30 ↑0.015 / 0.082 / 0.1920.012 / 0.063 / 0.171
Joint pose AUC @5 / 15 / 30 ↑0.003 / 0.057 / 0.1660.002 / 0.045 / 0.153

GT vs prediction, four validation scenes

Discussion of held-out errors

3.3 Bonus: Mini-VGGT Exploration 10 pts

Describe your question, experimental settings, results and conclusions, with plots or screenshots, possible explanations and limitations.

Question: direct point-map prediction vs camera + depth

The baseline reconstructs 3D points by unprojecting predicted depth with predicted cameras. I added a head that predicts canonical world-space points directly, trained it jointly with the camera and depth heads, and asked two things: (1) which point map is more accurate, and (2) does the extra point supervision change camera and depth quality (multi-task effect)?

Implementation

python -m models.vggt.train --config configs/vggt_points.json --output outputs/vggt_points
python scripts/compare_points.py --checkpoint outputs/vggt_points/best.pt --split {val,test} --eval-root data/scannetpp/prepared_eval \
    --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825

Results: point-map accuracy

Mean L2 point error, normalized units (meters in parentheses). Lower is better.

Points fromBaseline (Q3.2), val+ point head, valBaseline (Q3.2), test+ point head, test
Point head (direct)—0.265 (0.549 m)—0.242 (0.470 m)
Predicted cameras + predicted depth0.248 (0.501 m)0.260 (0.533 m)0.251 (0.491 m)0.231 (0.444 m)
GT cameras + predicted depth0.128 (0.281 m)0.136 (0.301 m)0.142 (0.287 m)0.146 (0.288 m)

Results: effect on camera and depth

MetricBaseline, val+ points, valBaseline, test+ points, test
Depth AbsRel ↓0.1270.1390.1510.151
Depth δ1 ↑0.8610.8440.8210.808
Rotation error ↓12.24°12.19°10.92°10.30°
Translation error ↓42.43°42.51°42.02°38.45°
Rotation AUC@30 ↑0.6960.7020.6740.705
Translation AUC@30 ↑0.1920.1840.1710.194
Pose AUC@5 / 15 / 30 ↑0.003 / 0.057 / 0.1660.007 / 0.061 / 0.1670.002 / 0.045 / 0.1530.004 / 0.059 / 0.179
Loss curves with the point head; the total includes the point term, and validation camera loss falls faster than in the baseline
Figure 28. Losses with the point head (total includes the point term). Compare the camera panel with Figure 14: validation camera loss reaches 0.15 by epoch ~100 here vs ~epoch 250 in the baseline.

GT for these scenes is in Figures 15 and 19.

Conclusions