16-825 Learning for 3D Vision
Shobhit Arora | Andrew ID: shobhit2
We unproject depth pixels into 3D using the camera intrinsics and transform the points into world coordinates. We then canonicalize the reconstruction relative to the first camera, obtaining a consistent reference frame and normalized scale. The following figures show the geometry for two scenes.
Each pixel ray is represented using a camera origin and a unit direction expressed in the common world frame. Its Plücker representation is (d, m), where d is the ray direction and m = o × d is the moment. The moment encodes the position of the ray relative to the coordinate origin.
The provided data-check script completed successfully. Depth validity was approximately 98-99%, the point reprojection error was around 0.0001 pixels, and the canonicalized centroid distance was approximately 1.
| Check | Result |
|---|---|
| Training samples | 224 |
| Valid depth fraction | ~98-99% |
| Reprojection error | ~0.0001 pixels |
| Canonicalized centroid distance | ~1.0 |
The geometry validation checks passed, indicating consistent depth unprojection, camera transformations, and scene normalization.
The Mini-LVSM model was trained to overfit a fixed batch for 2,000 optimization steps. It achieved a validation PSNR of approximately 35.84 dB, demonstrating that the implementation can memorize the selected views.
The model was trained for 100 epochs. The configuration used 8 transformer layers, a model width of 384, and patch size 16. The learned model was evaluated using different numbers of source views and camera-ray conditions.
| Source Views | Correct Rays PSNR (dB) | Observation |
|---|---|---|
| 1 | 15.05 | Lowest of the three |
| 2 | 15.92 | Improved |
| 4 | 16.52 | Highest of the three |
Increasing the number of available source views improved reconstruction PSNR in this experiment. Incorrectly shuffled or removed ray conditioning substantially reduced performance, supporting the importance of geometric information for novel-view synthesis.
| Source Views | Condition | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | Correct | 15.05 | 0.5562 |
| 1 | Shuffled | 11.75 | 0.4876 |
| 1 | None | 12.47 | 0.4992 |
| 2 | Correct | 15.92 | 0.5570 |
| 2 | Shuffled | 11.45 | 0.4728 |
| 2 | None | 12.43 | 0.4974 |
| 4 | Correct | 16.52 | 0.5698 |
| 4 | Shuffled | 11.60 | 0.4749 |
| 4 | None | 12.43 | 0.5051 |
We investigated whether fine-tuning with different numbers of source views improves generalization to other view counts. Starting from the same full-training checkpoint, separate models were fine-tuned using 1, 2, 4, or 6 source views.
Validation PSNR (dB):
| Train S / Eval S | 1 | 2 | 4 | 6 |
|---|---|---|---|---|
| 1 | 15.04 | 15.80 | 16.29 | 16.11 |
| 2 | 15.06 | 15.92 | 16.60 | 16.36 |
| 4 | 14.99 | 15.89 | 16.68 | 16.53 |
| 6 | 14.98 | 15.89 | 16.73 | 16.61 |
Fine-tuning with more source views generally improved PSNR when evaluating with four or six source views, but improvements were modest. Performance depended on the evaluation view count, and comparisons across columns should be interpreted carefully because the target-view partitions differ.
| Train S / Eval S | 1 | 2 | 4 | 6 |
|---|---|---|---|---|
| 1 | 0.5596 | 0.5572 | 0.5649 | 0.5258 |
| 2 | 0.5568 | 0.5574 | 0.5704 | 0.5333 |
| 4 | 0.5505 | 0.5520 | 0.5691 | 0.5357 |
| 6 | 0.5483 | 0.5505 | 0.5694 | 0.5359 |
Mini-VGGT was trained on one fixed scene for 40 epochs. After correcting the alternating frame/global attention implementation, the network successfully overfit both camera poses and depth.
| Metric | Final Result |
|---|---|
| Rotation error | 0.126° |
| Translation-direction error | 0.299° |
| Depth AbsRel | 0.0365 |
| Depth δ1 | 0.9554 |
| Joint pose AUC @30° | 0.9964 |
The model was trained for 1,000 epochs on the ScanNet++ training split. The final evaluations use the checkpoint selected by validation performance.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1402 | 0.1542 |
| Depth normalized RMSE ↓ | 0.1852 | 0.2005 |
| Depth δ1 ↑ | 0.8422 | 0.8138 |
| Rotation error ↓ | 12.40° | 10.03° |
| Translation error ↓ | 43.60° | 40.73° |
| Split / Metric | @5° | @15° | @30° |
|---|---|---|---|
| Val Rotation | 0.1839 | 0.4992 | 0.6935 |
| Val Translation | 0.0122 | 0.0804 | 0.1807 |
| Val Joint Pose | 0.0031 | 0.0576 | 0.1570 |
| Test Rotation | 0.1923 | 0.5223 | 0.7125 |
| Test Translation | 0.0099 | 0.0727 | 0.1796 |
| Test Joint Pose | 0.0008 | 0.0489 | 0.1607 |
The model achieves stronger depth and rotation performance than translation-direction accuracy on held-out scenes. Low translation and joint-pose AUC values indicate that camera translation remains a limitation.
An additional point prediction head was trained to predict dense world-space XYZ coordinates. The model was initialized from the trained Mini-VGGT checkpoint and fine-tuned with point supervision and encoder gradients enabled.
| Metric | Final Result |
|---|---|
| Points L2 (normalized) | 0.6330 |
| Points L2 (meters) | 1.2867 m |
| Depth AbsRel | 0.1348 |
| Depth δ1 | 0.8587 |
| Rotation error | 11.89° |
| Translation error | 43.18° |
The following comparisons show ground-truth geometry and camera/depth-based reconstructions from the point-supervised model.
We added a dense world-space point prediction head and fine-tuned the model for 30 epochs, followed by a separate 70-epoch extension. Point prediction error improved, but the direct PointHead output remained spatially scattered and did not consistently preserve coherent scene geometry. The validation loss gradually decreases during the additional 70-epoch fine-tuning stage, while training loss exhibits substantial fluctuations. Despite these fluctuations, the model achieves modest improvements in point prediction and depth estimation.