16-825 Assignment 4: Multi-View Transformers

Andrew ID: skorada

1. Multi-View Data Preprocessing

Canonicalization transforms every view into the first camera's frame and divides translations, depth, and points by one shared scene scale. Camera 0 is identity, and the valid-point centroid is one unit from the origin.

1.1 Unprojection

Unprojected ScanNet++ points and camera centers in the original world frame
Original world-frame points and cameras, before canonicalization.

1.2 Canonicalization

Canonicalized ScanNet++ points, camera zero, and valid-point centroid
Canonical camera frame. Camera 0 is at the origin and the displayed valid-point centroid distance is 1.003 (the full-resolution check reports 1.000).

1.3 Data validation

sequence                              views  valid depth  point reproj.  ray reproj.   centroid
e050c15a8d_frame_007080                 8       0.9779       0.000110      0.000094      1.000000
5f99900f09_frame_002820                 8       0.9903       0.000111      0.000111      1.000000
47b37eb6f9_frame_002720                 8       0.9610       0.000094      0.000089      1.000000
4c5c60fa76_frame_006690                 8       0.7517       0.000107      0.000094      1.000000

Batch shapes: rgb (2,8,3,224,224), depth (2,8,1,224,224), K (2,8,3,3),
poses (2,8,4,4), points/ray origins/ray directions (2,8,3,224,224),
Plücker rays (2,8,6,224,224).

1.4 Geometry explanation

World-space ray directions are obtained by applying the camera-to-world rotation to each normalized camera ray; translation does not affect a direction. Each ray is represented by Plücker coordinates [d, m], where d is the unit direction and m = o × d is the moment of the ray about the world origin.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 30.84 dB.

Mini-LVSM overfit source views, ground truth, and predictions
Overfit visualization: source views, ground truth, and predicted targets.

2.2 Training and Evaluation

I trained for 1,000 epochs with 2 source views, batch size 4, 8×8 patches, width 384, 8 transformer layers, learning rate 2×10−4, weight decay 0.05, 200 warmup steps, cosine decay, and bfloat16 autocast. The best checkpoint was epoch 969 with validation PSNR 17.70 dB and SSIM 0.621.

Mini-LVSM training and validation loss, PSNR, and SSIM curves
Mini-LVSM training curves.
Mini-LVSM training visualization
Epoch 975 validation visualization.

Source-view and ray-conditioning ablations

Source viewsConditioningPSNR ↑SSIM ↑
1Correct16.210.6015
1Shuffled11.560.4766
1None12.440.5252
2Correct17.700.6206
2Shuffled11.150.4533
2None12.430.5234
4Correct19.030.6472
4Shuffled11.220.4471
4None12.400.5324

Discussion: With correct rays, increasing sources from 1 to 4 adds 2.82 dB PSNR and 0.0458 SSIM. Shuffled rays score 11.15–11.56 dB, below zeroed rays at about 12.4 dB, because they request the wrong viewpoint.

2.3 Bonus: L1-augmented reconstruction

Question: Does adding a small L1 term improve structural similarity and sharpness without sacrificing PSNR? Both runs use the same architecture, split, seed, optimizer, and 1,000-epoch schedule; only the objective changes from MSE to MSE + 0.1 × L1.

ObjectiveValidation PSNR ↑Validation SSIM ↑
MSE baseline17.6970.6206
MSE + 0.1 L117.7480.6262
Validation PSNR and SSIM curves comparing MSE against MSE plus L1
Controlled validation comparison over 1,000 epochs.
L1-augmented Mini-LVSM validation predictions
L1-augmented validation predictions at epoch 975.

Conclusion: The L1 term produced a small but consistent improvement: +0.051 dB PSNR and +0.0055 SSIM. The gain is small relative to epoch variation; repeated seeds and a perceptual metric are needed to support a sharpness claim.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Mini-VGGT overfit total, camera, and depth loss curves
Overfit total, camera, and depth losses.
Mini-VGGT overfit ground-truth and predicted geometry
Single-clip overfit geometry comparison.

3.2 Training and Evaluation

I trained on 8 views sampled from a 32-frame window at 224×224 resolution, batch size 16, for 1,000 epochs. The model uses width 384, 8 frame/global attention pairs, 6 heads, frozen DINOv2 ViT-B/14 features, learning rate 6×10−4, weight decay 0.01, and bfloat16 precision.

Mini-VGGT full-training total, camera, and depth loss curves
Normal-training total, camera, and depth loss curves.

Validation and test metrics

SplitAbsRel ↓δ1 ↑Rot. AUC 5° / 15° / 30° ↑Trans. AUC 5° / 15° / 30° ↑Joint AUC 5° / 15° / 30° ↑
Validation0.15660.80090.1913 / 0.5107 / 0.70020.0099 / 0.0794 / 0.17700.0046 / 0.0603 / 0.1571
Test0.16530.78690.1796 / 0.5087 / 0.71120.0077 / 0.0536 / 0.15570.0018 / 0.0348 / 0.1418

Held-out geometry

Error analysis: Large surfaces and room layout are recovered, but fine geometry is smoothed and some view boundaries separate. Rotation is stronger than translation (validation AUC@30° 0.700 versus 0.177); failures concentrate around occlusions, thin or textureless structures, and limited-parallax views.

3.3 Bonus: Direct point-map prediction

Question: Can direct world-point prediction reduce the view-boundary errors from separately predicted cameras and depth? The bonus adds an RGB-refined three-channel decoder and masked, trimmed Smooth L1 point loss with weight 1. Both models use the same data, architecture, optimizer, and 1,000-epoch schedule; the bonus checkpoint is epoch 858.

Validation modelAbsRel ↓δ1 ↑Rot. AUC 5° / 15° / 30° ↑Trans. AUC 5° / 15° / 30° ↑Joint AUC 5° / 15° / 30° ↑Direct-point L2 ↓
Baseline0.15660.80090.1913 / 0.5107 / 0.70020.0099 / 0.0794 / 0.17700.0046 / 0.0603 / 0.1571—
Joint + point head0.14160.82340.1888 / 0.5156 / 0.69910.0122 / 0.0784 / 0.19540.0046 / 0.0563 / 0.16800.3122
Camera, depth, point, and total VGGT bonus training losses
Joint camera, depth, and direct-point training over 1,000 epochs.
Ground truth, camera plus depth reconstruction, and direct point-head prediction
Held-out scene: ground truth, camera-plus-depth reconstruction, and direct point-map prediction from the same best checkpoint.

Conclusion: joint point supervision improved depth AbsRel by 9.5%, δ1 by 0.0225, translation AUC@30° by 0.0184, and joint pose AUC@30° by 0.0110. Rotation AUC@30° stayed unchanged. The direct cloud captures room layout but smooths fine structure. One run and one point-loss weight cannot separate multi-task benefit from seed variance or establish the best weighting.