Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision

Shobhit Arora | Andrew ID: shobhit2

Part 1 Multi-View Data Preprocessing

Depth Unprojection and Camera Canonicalization

We unproject depth pixels into 3D using the camera intrinsics and transform the points into world coordinates. We then canonicalize the reconstruction relative to the first camera, obtaining a consistent reference frame and normalized scale. The following figures show the geometry for two scenes.

Scene 1

Scene 1 before canonicalization
Scene 1: Depth unprojection.
Scene 1 after canonicalization
Scene 1: Canonicalized reconstruction.

Scene 2

Scene 2 before canonicalization
Scene 2: Depth unprojection.
Scene 2 after canonicalization
Scene 2: Canonicalized reconstruction.

Camera Rays and Plücker Coordinates

Each pixel ray is represented using a camera origin and a unit direction expressed in the common world frame. Its Plücker representation is (d, m), where d is the ray direction and m = o × d is the moment. The moment encodes the position of the ray relative to the coordinate origin.

Data Validation

The provided data-check script completed successfully. Depth validity was approximately 98-99%, the point reprojection error was around 0.0001 pixels, and the canonicalized centroid distance was approximately 1.

Geometry Validation Results

CheckResult
Training samples224
Valid depth fraction~98-99%
Reprojection error~0.0001 pixels
Canonicalized centroid distance~1.0

The geometry validation checks passed, indicating consistent depth unprojection, camera transformations, and scene normalization.

Part 2 Novel-View Synthesis with Mini-LVSM

Q2.1 Single-Batch Overfitting

The Mini-LVSM model was trained to overfit a fixed batch for 2,000 optimization steps. It achieved a validation PSNR of approximately 35.84 dB, demonstrating that the implementation can memorize the selected views.

Mini-LVSM overfitting
Mini-LVSM overfitting results (PSNR: 35.84 dB, SSIM: 0.9292).

Q2.2 Full Training and Ablations

The model was trained for 100 epochs. The configuration used 8 transformer layers, a model width of 384, and patch size 16. The learned model was evaluated using different numbers of source views and camera-ray conditions.

Training and Validation Curves

Mini-LVSM training and validation loss
Training and validation loss over 100 epochs.
Mini-LVSM training and validation PSNR
Training and validation PSNR over 100 epochs.
Source Views Correct Rays PSNR (dB) Observation
115.05Lowest of the three
215.92Improved
416.52Highest of the three

Increasing the number of available source views improved reconstruction PSNR in this experiment. Incorrectly shuffled or removed ray conditioning substantially reduced performance, supporting the importance of geometric information for novel-view synthesis.

Training Progression

Epoch 0
Epoch 0
Epoch 50
Epoch 50
Epoch 95
Epoch 95

Ray-Conditioning Ablation

Source ViewsCondition PSNR (dB)SSIM
1Correct15.050.5562
1Shuffled11.750.4876
1None12.470.4992
2Correct15.920.5570
2Shuffled11.450.4728
2None12.430.4974
4Correct16.520.5698
4Shuffled11.600.4749
4None12.430.5051

Effect of Camera-Ray Conditioning (S = 2)

Correct rays
Correct camera rays
Shuffled rays
Shuffled camera rays
No rays
No camera-ray conditioning

Effect of Source-View Count

One source view
1 source view
Two source views
2 source views
Four source views
4 source views

Q2.3 Bonus: Training Source-View Count

We investigated whether fine-tuning with different numbers of source views improves generalization to other view counts. Starting from the same full-training checkpoint, separate models were fine-tuned using 1, 2, 4, or 6 source views.

Validation PSNR (dB):

Train S / Eval S12 46
115.0415.8016.2916.11
215.0615.9216.6016.36
414.9915.8916.6816.53
614.9815.8916.7316.61

Fine-tuning with more source views generally improved PSNR when evaluating with four or six source views, but improvements were modest. Performance depended on the evaluation view count, and comparisons across columns should be interpreted carefully because the target-view partitions differ.

SSIM Comparison

Train S / Eval S 1246
10.55960.55720.56490.5258
20.55680.55740.57040.5333
40.55050.55200.56910.5357
60.54830.55050.56940.5359

Qualitative Comparison (S_eval = 4)

Train S1
Fine-tuned with 1 source view
Train S2
Fine-tuned with 2 source views
Train S4
Fine-tuned with 4 source views
Train S6
Fine-tuned with 6 source views

Part 3 Multi-View Geometry with Mini-VGGT

Q3.1 Single-Scene Overfitting

Mini-VGGT was trained on one fixed scene for 40 epochs. After correcting the alternating frame/global attention implementation, the network successfully overfit both camera poses and depth.

MetricFinal Result
Rotation error0.126°
Translation-direction error0.299°
Depth AbsRel0.0365
Depth δ10.9554
Joint pose AUC @30°0.9964

Overfitting Loss Curve

Mini-VGGT single-scene overfitting.

Q3.2 Full Training and Evaluation

The model was trained for 1,000 epochs on the ScanNet++ training split. The final evaluations use the checkpoint selected by validation performance.

MetricValidationTest
Depth AbsRel ↓0.14020.1542
Depth normalized RMSE ↓0.18520.2005
Depth δ1 ↑0.84220.8138
Rotation error ↓12.40°10.03°
Translation error ↓43.60°40.73°

Pose AUC

Split / Metric@5° @15°@30°
Val Rotation0.18390.49920.6935
Val Translation0.01220.08040.1807
Val Joint Pose0.00310.05760.1570
Test Rotation0.19230.52230.7125
Test Translation0.00990.07270.1796
Test Joint Pose0.00080.04890.1607

The model achieves stronger depth and rotation performance than translation-direction accuracy on held-out scenes. Low translation and joint-pose AUC values indicate that camera translation remains a limitation.

Training Loss Curve

Mini-VGGT full training.

Ground Truth vs. Predicted Geometry

Scene 1

Scene 2

Scene 3

Scene 4

Q3.3 Bonus: World-Space Point Prediction

An additional point prediction head was trained to predict dense world-space XYZ coordinates. The model was initialized from the trained Mini-VGGT checkpoint and fine-tuned with point supervision and encoder gradients enabled.

MetricFinal Result
Points L2 (normalized)0.6330
Points L2 (meters)1.2867 m
Depth AbsRel0.1348
Depth δ10.8587
Rotation error11.89°
Translation error43.18°

Qualitative Reconstructions

The following comparisons show ground-truth geometry and camera/depth-based reconstructions from the point-supervised model.

Scene 1

Scene 2

Fine-Tuning Loss Curve

Additional 70-epoch PointHead fine-tuning stage.

We added a dense world-space point prediction head and fine-tuned the model for 30 epochs, followed by a separate 70-epoch extension. Point prediction error improved, but the direct PointHead output remained spatially scattered and did not consistently preserve coherent scene geometry. The validation loss gradually decreases during the additional 70-epoch fine-tuning stage, while training loss exhibits substantial fluctuations. Despite these fluctuations, the model achieves modest improvements in point prediction and depth estimation.