16-825: Learning for 3D Vision, Assignment 4¶
Multi-View Transformers
Kaushik Chavali (kchavali)
Contents: Part 1: Multi-view data · Part 2: Mini-LVSM (Q2.1, Q2.2, Q2.3 bonus) · Part 3: Mini-VGGT (Q3.1, Q3.2, Q3.3 bonus)
Part 1. Multi-View Data Preprocessing¶
After unprojection (Q1.2)¶
Points in the original ScanNet++ world frame, in meters. The first camera (red) sits at an arbitrary location.

Valid-point centroid distance: 6.668 m

Valid-point centroid distance: 4.435 m
After canonicalization (Q1.3)¶
The same frames in the first camera's coordinates, scaled so the centroid is 1 unit away.

Valid-point centroid distance: 1.000 normalized units
Scene scale: 1.867 m

Valid-point centroid distance: 1.000 normalized units
Scene scale: 2.830 m
Final check (Q1.7)¶
check_data.py output on four training samples. Reprojection errors are about 10-4 px (limit 0.01 px).
| Sample | Views | Resolution | Valid depth | Max point reproj. error (px) | Max ray reproj. error (px) | Centroid distance |
|---|---|---|---|---|---|---|
e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 1.10e-04 | 9.41e-05 | 1.0000 |
5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.11e-04 | 1.11e-04 | 1.0000 |
47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 9.59e-05 | 8.90e-05 | 1.0000 |
4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.03e-04 | 1.01e-04 | 1.0000 |
Raw check_data.py output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010268093319609761,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Explanations¶
Canonicalization. All poses are multiplied by the inverse of camera 0's pose, so camera 0 becomes the identity. Translations, depth and points are then divided by a shared scale that puts the valid-point centroid 1 unit from camera 0. This removes the arbitrary world frame and room size, so every sample uses consistent coordinates.
World-space ray directions. K-1[u, v, 1]T gives a pixel's direction in camera coordinates. Rotating it by the camera-to-world R and normalizing gives a unit direction in the shared world frame, so rays from different views can be compared. The ray origin is the camera center.
Plücker moment. A ray is stored as [d, m] with m = o × d. Any point on the line gives the same m, since (o + s·d) × d = o × d. The moment adds the position information that d lacks: parallel rays from different cameras share d but have different m.
Part 2. Novel-View Synthesis with Mini-LVSM¶
2.1. Overfitting¶
Columns are source, source, ground truth, prediction. The prediction recovers the layout, colors and most edges, so the model, loss and training step work.
val_epoch000.png2.2. Training and Evaluation¶
Training settings¶
| Setting | Value |
|---|---|
| Config | configs/lvsm.json, unchanged |
| Source views during training | S = 2 (the other 6 of 8 sampled views are targets) |
| Model | patch size 8, width 384, 8 transformer layers, 14.6M parameters |
| Data | 224×224, sampling range 32, 224 training scenes |
| Optimization | batch 4, 100 epochs (56 steps per epoch), AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps, gradient clip 1.0 |
| Precision and time | bfloat16 autocast with loss in float32, about 24 s per epoch (40 min total) |
Training curves¶
Validation PSNR and SSIM for each ablation¶
| Source views | Target rays | PSNR (dB) | SSIM | Images |
|---|---|---|---|---|
| 1 | correct | 15.20 | 0.558 | 196 |
| 1 | shuffled | 11.82 | 0.487 | 196 |
| 1 | none | 12.47 | 0.530 | 196 |
| 2 | correct | 16.15 | 0.560 | 168 |
| 2 | shuffled | 11.49 | 0.472 | 168 |
| 2 | none | 12.45 | 0.529 | 168 |
| 4 | correct | 16.89 | 0.575 | 112 |
| 4 | shuffled | 11.60 | 0.473 | 112 |
| 4 | none | 12.47 | 0.538 | 112 |
Visualizations¶
Columns are the source views, then ground truth, then prediction. Each row is one target view.
During training (S = 2):
Evaluation of best.pt, all nine settings:
Discussion¶
Source views. With correct rays, PSNR rises from 15.20 dB at S=1 to 16.15 dB at S=2 and 16.89 dB at S=4, since more views cover more of the scene. The number of targets also falls as S grows (196, 168, 112 images), so the settings are not scored on identical targets.
Ray conditioning. Correct rays beat shuffled rays by 3.4 to 5.3 dB. With no rays the model outputs a nearly uniform gray-beige image and stays near 12.5 dB for any S. Shuffled rays score lower still because they put plausible structure in the wrong place, which MSE penalizes more than a flat color.
Quality. Predictions get the coarse layout and colors right but are blurry, with visible patch-sized blocks. Validation PSNR was still rising slowly at epoch 100, with no sign of overfitting.
2.3. Bonus: Loss Function¶
Question¶
How do PSNR, SSIM and visual sharpness change when an SSIM term is added to the MSE loss?
Experimental settings¶
The loss is MSE + 0.3·(1 − SSIM). Everything else matches Q2.2, and best.pt is selected by validation PSNR in both runs.
Results¶
| Source views | Target rays | PSNR: MSE | PSNR: MSE + SSIM | SSIM: MSE | SSIM: MSE + SSIM |
|---|---|---|---|---|---|
| 1 | correct | 15.20 | 14.78 | 0.558 | 0.577 |
| 1 | shuffled | 11.82 | 11.90 | 0.487 | 0.518 |
| 1 | none | 12.47 | 12.44 | 0.530 | 0.546 |
| 2 | correct | 16.15 | 15.41 | 0.560 | 0.580 |
| 2 | shuffled | 11.49 | 11.68 | 0.472 | 0.511 |
| 2 | none | 12.45 | 12.42 | 0.529 | 0.542 |
| 4 | correct | 16.89 | 15.78 | 0.575 | 0.593 |
| 4 | shuffled | 11.60 | 11.84 | 0.473 | 0.519 |
| 4 | none | 12.47 | 12.47 | 0.538 | 0.553 |
Conclusions, explanations and limitations¶
Metrics. SSIM improved in all nine settings, by 0.014 to 0.046. With correct rays, PSNR dropped by 0.42 to 1.11 dB, which makes sense because MSE is no longer the only objective.
Sharpness. The SSIM images are smoother, with less patch blocking, but not sharper: no new detail appears and colors are less saturated. A smooth image with roughly correct local contrast can score well on SSIM without having fine detail.
Limitations. Each model was trained once with one SSIM weight, and metrics were still rising at epoch 100. The loss values use different definitions, so only PSNR and SSIM are compared.
Part 3. Multi-View Geometry with Mini-VGGT¶
3.1. Overfitting¶
40 epochs of 25 steps on one fixed scene. All three losses decrease. Total and depth loss go below zero because the confidence term includes −0.2·log(c).
| Depth AbsRel | Depth δ1 | Mean rotation error | Mean translation direction error | Joint pose AUC@30° |
|---|---|---|---|---|
| 0.037 | 0.954 | 0.23° | 0.36° | 0.998 |
The prediction closely matches ground truth, apart from blockier depth and slightly noisier surfaces.
3.2. Training and Evaluation¶
Training settings¶
| Setting | Value |
|---|---|
| Config | configs/vggt_4070.json: configs/vggt.json with the two changes below |
| Batch size | 8 (default 16). A test at batch 12 ran out of memory on the 12 GB GPU; batch 8 uses about 8 GB |
| Learning rate | 4e-4 (default 6e-4), lowered along with the batch size |
| Views | num_views = 8, sampling_range = 32 (required values, unchanged) |
| Model | frozen DINOv2 ViT-B/14, 8 frame/global attention pairs, width 384, 6 heads |
| Optimization | 1000 epochs (28 steps per epoch), AdamW, weight decay 0.01, 10 warmup epochs, cosine decay to 5% of the lr, gradient clip 1.0 |
| Loss weights | camera 5.0, depth 1.0 |
| Time and selection | about 4.1 hours; best.pt selected by validation loss (epoch 848) |
Loss curves¶
All three losses decrease. The one-epoch spike at epoch 265 (validation loss 5.50, mostly camera loss) dropped back to 1.19 the next epoch and did not affect later training. I did not investigate its cause.
Validation and test metrics¶
Evaluated with 8 views from the first 32 prepared frames, 224 px and seed 825, on the full splits (28 scenes each).
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel (lower is better) | 0.135 | 0.157 |
| Depth δ1 | 0.842 | 0.807 |
| Rotation AUC@5° | 0.202 | 0.213 |
| Rotation AUC@15° | 0.519 | 0.513 |
| Rotation AUC@30° | 0.706 | 0.703 |
| Translation AUC@5° | 0.020 | 0.008 |
| Translation AUC@15° | 0.074 | 0.063 |
| Translation AUC@30° | 0.185 | 0.175 |
| Joint pose AUC@5° | 0.005 | 0.004 |
| Joint pose AUC@15° | 0.052 | 0.046 |
| Joint pose AUC@30° | 0.160 | 0.154 |
| Mean rotation error (°) | 12.31 | 10.05 |
| Mean translation direction error (°) | 42.08 | 38.64 |
GT vs. predicted geometry on four validation scenes¶
Scene 09c1414f1b
21d970d8deScene 280b83fcf3Scene 2b1dc6d6a5
Discussion of errors on held-out scenes¶
Depth is the strongest result: about 84% of validation and 81% of test pixels are within a factor of 1.25 of ground truth. Predicted depth maps are blocky at the 14 px patch scale.
Rotation is moderate: mean error is 12.3° on validation and 10.0° on test. AUC is about 0.70 at 30° but only about 0.2 at 5°, so coarse orientation is learned but precise alignment is not.
Translation direction is the weak point, with about 40° mean error, which holds joint pose AUC down. Small baselines may be one reason, since a small position error then becomes a large angle error.
Point clouds keep the room layout and camera path, but flat walls become thick, bowed surfaces and thin objects merge (the table and chairs in 09c1414f1b).
Generalization. The overfit run reached 0.2° rotation error against 10 to 12° here, so the main gap is generalizing to new scenes. Validation and test results are close overall.
3.3. Bonus: Point-Map Prediction¶
Question¶
Is directly predicting world-space points better than unprojecting predicted depth with predicted cameras? Does point supervision change depth and camera accuracy?
Experimental settings¶
PointHead follows the depth head design: a linear layer to 3 channels per pixel plus RGB refinement, with no output activation since coordinates can be negative. The loss is the mean per-pixel Euclidean error plus the depth gradient loss on x, y and z, with weight 1.0. Everything else matches Q3.2 (about 4.8 hours). The provided evaluation only scores the direct head, so scripts/compare_points.py applies the same metric to unprojected points.
Results¶
Point error (mean L2 distance to ground truth over valid pixels, normalized units, lower is better):
| Method | Validation | Test |
|---|---|---|
| Baseline (Q3.2): unprojected depth + camera | 0.273 | 0.251 |
| Point-head model: unprojected depth + camera | 0.245 | 0.203 |
| Point-head model: direct point head | 0.255 | 0.213 |
Depth and camera metrics, baseline against point-head model:
| Metric | Val: baseline | Val: + point head | Test: baseline | Test: + point head |
|---|---|---|---|---|
| Depth AbsRel (lower is better) | 0.135 | 0.131 | 0.157 | 0.143 |
| Depth δ1 | 0.842 | 0.845 | 0.807 | 0.844 |
| Mean rotation error (°) | 12.31 | 11.31 | 10.05 | 9.15 |
| Mean translation direction error (°) | 42.08 | 41.33 | 38.64 | 37.71 |
| Joint pose AUC@5° | 0.005 | 0.008 | 0.004 | 0.002 |
| Joint pose AUC@15° | 0.052 | 0.058 | 0.046 | 0.061 |
| Joint pose AUC@30° | 0.160 | 0.160 | 0.154 | 0.191 |
| Rotation AUC@30° | 0.706 | 0.705 | 0.703 | 0.743 |
| Translation AUC@30° | 0.185 | 0.182 | 0.175 | 0.209 |
Conclusions, explanations and limitations¶
Direct vs. unprojected. Within the same model, unprojection beats the direct head by 3.7% on validation and 4.8% on test. Unprojection splits the problem into per-view depth and pose, while the direct head must also account for camera motion, which may explain the gap. Visually, both recover the same coarse layout with bowed walls.
Point supervision. Compared with the baseline, unprojected point error fell by 10.0% on validation and 19.0% on test. Depth and mean pose errors improved on both splits, most on test (AbsRel 0.157 to 0.143, joint pose AUC@30° 0.154 to 0.191). Joint pose AUC@30° on validation is essentially unchanged.
Limitations. There is one training run per model and 28 scenes per split, so small differences may be noise. The point head also has no learned confidence, unlike depth, which may disadvantage it.



































