Multi-View Transformers¶
16-825 Learning for 3D: Assignment 4
Author: Benny Zhu (haozhenz)
Date: November 6th 2026
1.3 Camera and Depth Canonicalization¶
python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization
Canonicalized geometry (camera 0 = identity):
| Scene 1 |
|---|
![]() |
| Scene | Camera 0 center | Valid-point centroid distance |
|---|---|---|
| Scene 1 | (0, 0, 0) | 1.0 |
1.7 Final Check¶
python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
Output:
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization: Express all camera poses relative to a reference frame. Typically the first camera is placed at the origin with identity rotation, and every other camera is rewritten relative to it, often with scale normalized as well. The model then sees only relative camera geometry, independent of the scene’s absolute position and scale.
World-space ray directions: For each pixel (u, v), unproject with the inverse intrinsics to get a camera-frame direction K⁻¹[u, v, 1]ᵀ, rotate it into the world frame with the camera-to-world rotation R, and normalize: d = normalize(R K⁻¹[u, v, 1]ᵀ). This is the direction of the ray leaving the camera center through that pixel.
Plücker moment: With camera center o, the moment is m = o × d. Together, (d, m) is a 6D representation of the ray. Replacing o with any other point on the ray leaves m unchanged, so the pair identifies the line itself: d encodes its orientation and m its offset from the origin.
2.2 Training and Evaluation (30 points)¶
python -m models.lvsm.train --config configs/lvsm.json
python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none
Training Settings¶
| Setting | Value |
|---|---|
num_source_views |
2 |
| Num views / sampling range | 8 / 32 |
| Batch size | 4 |
| Epochs | 100 |
| Learning rate | 2e-4 |
| Patch size / width / depth | 8 / 384 / 8 layers |
| AMP (bf16) | True |
Training Curves¶

Ablation Results (Validation)¶
| Source views | Conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | correct | 15.19246549022441 | 0.5581682409558978 |
| 1 | shuffled | 11.808103288922991 | 0.4875379484526965 |
| 1 | none | 12.474374420788823 | 0.5274129059849954 |
| 2 | correct | 16.12853331792922 | 0.5601188568841844 |
| 2 | shuffled | 11.480516342889695 | 0.47215169952029273 |
| 2 | none | 12.45132791428339 | 0.526719229561942 |
| 4 | correct | 16.854456220354354 | 0.5747556941849845 |
| 4 | shuffled | 11.587574277605329 | 0.47292416436331614 |
| 4 | none | 12.45746558053153 | 0.5379406639507839 |
Visualizations¶
Training:
Epoch 0:

Epoch 95:

Evaluation:
| Correct conditioning | Shuffled conditioning | No conditioning | |
|---|---|---|---|
| 1 source view | ![]() |
![]() |
![]() |
| 2 source views | ![]() |
![]() |
![]() |
| 4 source views | ![]() |
![]() |
![]() |
Discussion¶
Effect of the number of source views: Increasing the number of source views from 1 to 2 to 4 gives the model more overlapping observations of the scene, which reduces ambiguity about occluded or unseen regions and generally sharpen predictions, closer to GT.
Effect of ray-conditioning: With correct target rays, the model is told exactly which viewpoint to render and should reconstruct an image close to GT. With shuffled rays, the model is given a different view's camera pose, so a well-behaved model should render content resembling that wrong viewpoint rather than the true target. With no conditioning, the model has no information about which view to synthesize and tends to fall back to a blurry, low-frequency average of plausible content, since it cannot commit to any specific viewpoint.
3.2 Training and Evaluation (30 points)¶
python -m models.vggt.train --config configs/vggt.json --output outputs/vggt_train
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/metrics.json --visualization outputs/vggt_train/visualization/index.html
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split test --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/test_metrics.json
Training Settings¶
| Setting | Value |
|---|---|
num_views / sampling_range |
8 / 32 |
| Batch size | 16 |
| Epochs | 1000 |
| Learning rate | 6e-4 |
| Encoder | Frozen DINOv2 ViT-B/14 |
| Aggregator | 8 frame/global pairs, width 384, 6 heads |
| Loss weights (camera / depth) | camera=5.0,depth=1.0 |
Training Loss Curve (total / camera / depth)¶

Evaluation Metrics¶
| Metric | Val | Test |
|---|---|---|
| Sample counts | ||
| Num. samples (scenes) | 28 | 28 |
| Num. valid depth pixels | 10,347,720 | 10,517,881 |
| Num. pose pairs | 784 | 784 |
| Num. degenerate pose pairs | 0 | 0 |
| Loss | ||
| Total loss | 0.8234 | 0.7552 |
| Camera loss | 0.1081 | 0.0902 |
| Depth loss | 0.2829 | 0.3043 |
| Depth accuracy | ||
| AbsRel ↓ | 0.155 | 0.169 |
| RMSE (normalized) ↓ | 0.203 | 0.209 |
| δ1 ↑ | 0.800 | 0.767 |
| Raw pose error (mean, degrees) | ||
| Rotation error (°) ↓ | 12.32 | 8.82 |
| Translation error (°) ↓ | 39.84 | 41.48 |
| Rotation AUC ↑ | ||
| @3° | 0.091 | 0.115 |
| @5° | 0.178 | 0.210 |
| @15° | 0.508 | 0.547 |
| @30° | 0.699 | 0.747 |
| Translation AUC ↑ | ||
| @3° | 0.0085 | 0.0084 |
| @5° | 0.0156 | 0.0099 |
| @15° | 0.0880 | 0.0683 |
| @30° | 0.2082 | 0.1639 |
| Joint Pose AUC ↑ | ||
| @3° | 0.0000 | 0.0009 |
| @5° | 0.0018 | 0.0026 |
| @15° | 0.0645 | 0.0516 |
| @30° | 0.1815 | 0.1517 |
GT vs. Predicted Geometry (4 validation scenes)¶
| Scene | GT | Prediction |
|---|---|---|
| Scene 1 | ![]() |
![]() |
| Scene 2 | ![]() |
![]() |
| Scene 3 | ![]() |
![]() |
| Scene 4 | ![]() |
![]() |
Discussion¶
Across all four held-out validation scenes, the predicted camera layout closely matches GT. In Scenes 1, 3 and 4, the relative camera cluste is reconstructed in the same spatial arrangement as GT.
The point cloud geometry, however, degrades more visibly than the camera poses. In every scene, the predicted structure is noticeably noisier and less sharply defined than GT — flat surfaces and edges (e.g. the shelf unit in Scene 2, the desk in Scene 4) appear "smeared" or ballooned outward rather than forming crisp planar structure. Scene 4 shows the largest degradation: GT shows a well-defined desk/room boundary, while the prediction spreads into a much more diffuse, scattered gray cloud, even though the camera trajectory above it is still well localized. This matches the quantitative depth error (AbsRel ≈ 0.16–0.17, i.e. roughly 15–17% relative depth error) — a moderate but non-trivial error that manifests visually as exactly this kind of loss of fine geometric detail rather than gross failure.
Taken together, this suggests the model has learned to generalize the global scene structure and camera arrangement well, but still struggles to generalize precise, per-pixel depth to unseen scenes — likely because depth regression is a harder, more scene-specific task than relative camera pose, and the model was trained for a comparatively short schedule relative to the diversity of ScanNet++ scenes.
(Bonus) 2.3 Mini-LVSM Analysis and Exploration (10 points)¶
Question: Does adding an L1 and SSIM loss term on top of the baseline MSE photometric loss improve the PSNR, SSIM, and visual sharpness of Mini-LVSM's synthesized views?
Experimental settings: Train with the same configuration as Q2.2 (100 epochs, S=2 source views), but with --aux-loss "l1=0.1,ssim=0.1" enabled, then evaluate with the same source-view/conditioning ablation as Q2.2 and compare PSNR/SSIM/visual sharpness against the MSE-only baseline..
| Source views | Conditioning | Baseline PSNR | +L1+SSIM PSNR | Baseline SSIM | +L1+SSIM SSIM |
|---|---|---|---|---|---|
| 1 | correct | 15.19 | 14.76 | 0.5582 | 0.5719 |
| 2 | correct | 16.13 | 15.30 | 0.5601 | 0.5743 |
| 4 | correct | 16.85 | 15.55 | 0.5748 | 0.5870 |
| Baseline (S=2) | Variant (S=2) |
|---|---|
![]() |
![]() |
Conclusions: Adding L1 and SSIM loss terms alongside MSE consistently decreases PSNR while consistently increasing SSIM across every evaluated setting. This is a coherent trade-off: MSE is minimized by hedging pixel values toward a blurry average, which maximizes PSNR but produces soft, low-structure predictions; SSIM explicitly rewards local structural/contrast consistency rather than exact per-pixel values, so optimizing for it pulls the model away from the PSNR-optimal blurry solution toward sharper, more structured outputs. This illustrates that PSNR and structural quality are not the same objective, and improving one can come at the cost of the other.























