Assignment 4: Multi-View Transformers¶
Name: Chenzhe Zhang
Course: 16-825 Learning for 3D Vision
1.3. Camera and Depth Canonicalization¶
Geometry is expressed in the first camera's coordinate frame and uniformly scaled so that the valid-point centroid is one unit from the origin.
1.7. Final Check¶
Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
All four samples contain 8 views at 224×224. Reprojection errors are below 0.01 pixels, and centroid distances are approximately 1.
| Sample | Valid depth | Max point error (px) | Max ray error (px) | Centroid distance |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.9779 | 1.13e-4 | 1.09e-4 | 1.000 |
| 5f99900f09_frame_002820 | 0.9903 | 1.23e-4 | 1.11e-4 | 1.000 |
| 47b37eb6f9_frame_002720 | 0.9610 | 9.88e-5 | 8.90e-5 | 1.000 |
| 4c5c60fa76_frame_006690 | 0.7517 | 1.09e-4 | 1.03e-4 | 1.000 |
Canonicalization. Geometry is expressed in the first camera's frame, with translations, depth and points divided by a shared scale so the valid-point centroid lies one unit from the origin.
World-space ray directions. Rays start at camera centers, with unit directions $\mathbf{d}=\operatorname{normalize}(RK^{-1}[u,v,1]^T)$.
Plücker moment. The moment $\mathbf{m}=\mathbf{o}\times\mathbf{d}$ encodes the ray's position relative to the origin; together, direction and moment represent an oriented 3D line.
2.2. Training and Evaluation¶
Training¶
I trained the default Mini-LVSM for 100 epochs using 2 source views, 8 total views, 224×224 images, and batch size 4.
Training used AdamW with learning rate 0.0002, weight decay 0.05, 200 warmup steps, and BF16 mixed precision on one NVIDIA L40S. The checkpoint with the highest validation PSNR was selected.
Training Curves¶
Training Visualization¶
Epoch 0. Each row shows two source views, ground truth, and prediction.
Epoch 95
Validation Results¶
| Source views | Ray conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | Correct | 15.20 | 0.5576 |
| 1 | Shuffled | 11.79 | 0.4861 |
| 1 | None | 12.43 | 0.5286 |
| 2 | Correct | 16.17 | 0.5595 |
| 2 | Shuffled | 11.43 | 0.4705 |
| 2 | None | 12.40 | 0.5261 |
| 4 | Correct | 16.81 | 0.5742 |
| 4 | Shuffled | 11.56 | 0.4735 |
| 4 | None | 12.47 | 0.5376 |
Discussion¶
Correct ray conditioning achieves the highest PSNR and SSIM at every source-view count. Increasing source views from 1 to 4 improves the reported PSNR from 15.20 to 16.81 dB, although the evaluated target views also change with S. Predictions recover coarse scene structure but remain blurry and lack fine details.
3.2. Training and Evaluation¶
Training¶
Mini-VGGT was trained for 1000 epochs using 8 views per sample, 224×224 images, and batch size 16, with the DINOv2 encoder frozen.
Training used AdamW with learning rate 0.0006, weight decay 0.01, and BF16 mixed precision on one NVIDIA L40S. The checkpoint with the lowest validation loss was selected.
Loss Curves¶
Quantitative Results¶
δ1 and AUC values are reported on a 0–1 scale. Lower AbsRel and higher δ1/AUC are better.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1369 | 0.1512 |
| Depth δ1 ↑ | 0.8509 | 0.8030 |
| Rotation AUC@5° ↑ | 0.2069 | 0.2051 |
| Rotation AUC@15° ↑ | 0.5344 | 0.5351 |
| Rotation AUC@30° ↑ | 0.7211 | 0.7422 |
| Translation AUC@5° ↑ | 0.0122 | 0.0094 |
| Translation AUC@15° ↑ | 0.0802 | 0.0808 |
| Translation AUC@30° ↑ | 0.1896 | 0.2005 |
| Joint Pose AUC@5° ↑ | 0.0033 | 0.0031 |
| Joint Pose AUC@15° ↑ | 0.0591 | 0.0553 |
| Joint Pose AUC@30° ↑ | 0.1694 | 0.1841 |
Discussion¶
On held-out scenes, the model recovers coarse geometry, but point clouds contain local distortions and misaligned surfaces. Camera translation direction is less accurate than rotation, limiting joint pose accuracy. Depth accuracy is lower on the test set than on the validation set.