16-825: Learning for 3D Vision

Assignment 4

Fall 2026

Multi-View Transformers

Contents

1. Multi-View Data Preprocessing

1. Data Preprocessing

20 points

After unprojection (world frame, meters)

Unprojection 036bce3393
036bce3393_frame_004470
Unprojection 076c822ecc
076c822ecc_frame_002890

After canonicalization

Canonicalization 036bce3393
036bce3393_frame_004470. First camera center (0, 0, 0), centroid distance 1.000.
Canonicalization 076c822ecc
076c822ecc_frame_002890. First camera center (0, 0, 0), centroid distance 1.000.

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 0.00010764275066321716, "centroid_distance": 1.0},
    {"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
    {"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
    {"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00011280165199423209, "max_ray_reprojection_error_px": 0.00010225349979009479, "centroid_distance": 0.9999999403953552}
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Reprojection errors are about 1e-4 px (limit 0.01) and the centroid distance is 1.

Explanations

Canonicalization. All poses are expressed relative to camera 0, so camera 0 becomes the identity. Translations, depth and points are then divided by one scene scale, the distance from camera 0 to the centroid of the valid points. This way every sample has the same reference frame and roughly the same size, and the model does not have to guess the arbitrary world origin or the absolute scale.

World-space ray directions. Each pixel direction K⁻¹[u, v, 1] is rotated by the camera rotation and normalized. Using world space instead of camera space means rays from different views live in the same frame, so the model can tell how the cameras are posed relative to each other.

Plücker moment. m = o × d. The direction alone cannot separate parallel rays from different cameras; the moment encodes where the line is (its size is the distance from the origin to the line). It is the same for any point on the ray, so [d, m] describes the line itself.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

10 points

2000 steps on one fixed batch. Final PSNR: 31.00 dB (SSIM 0.872).

LVSM overfit
val_epoch000.png. Columns: source, source, ground truth, prediction.

2.2 Training and Evaluation

30 points

Settings. Default configs/lvsm.json: patch 8, width 384, 8 layers, 8 views per sample with 2 sources, batch size 4, lr 2e-4, 100 epochs, bf16. About 27 minutes on one A100.

LVSM training curves
Training curves
Source viewsConditioningPSNR ↑SSIM ↑
1correct15.200.5586
shuffled11.800.4875
none12.460.5292
2correct16.140.5603
shuffled11.490.4720
none12.440.5277
4correct16.870.5749
shuffled11.590.4729
none12.450.5378

Training visualizations

Epoch 0
Epoch 0
Epoch 95
Epoch 95

Evaluation visualizations

S=1 correct
S = 1, correct
S=2 correct
S = 2, correct
S=4 correct
S = 4, correct
S=2 shuffled
S = 2, shuffled
S=2 none
S = 2, none

Columns: source views, ground truth, prediction.

Discussion

Source views. More sources help: PSNR goes from 15.20 (1 view) to 16.14 (2) to 16.87 dB (4), because more of the target is visible in at least one source.

Ray conditioning. Correct rays are 3–5 dB better than shuffled or zero rays, so the model really uses the target camera. With zero rays it outputs a flat average color, and its score is about the same for any number of sources. Shuffled rays are even worse than zero rays, because the model confidently renders the wrong view.

Overall quality is still low. Predictions only get the rough layout and colors, and validation PSNR was still rising at epoch 100, so the model is undertrained.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

10 points

One fixed clip, 40 epochs × 25 steps, lr 1e-3. All losses go down. At the end, depth AbsRel is 0.037 and pose AUC@30 is 0.99. The total loss goes below zero because of the −0.2 log c confidence term.

Overfit loss curve
Overfit loss: total, camera, depth
Overfit GT
Ground truth
Overfit prediction
Prediction

3.2 Training and Evaluation

30 points

Settings. Default configs/vggt.json: frozen DINOv2 ViT-B/14, width 384, 8 frame/global pairs, 8 views, sampling range 32, batch size 16, lr 6e-4 with cosine decay over 1000 epochs. One shared A100. The GPU slowed down halfway through, so I stopped at epoch 722 (about 8.6 h). The results below use best.pt from epoch 721.

VGGT loss curve
Training loss: total, camera, depth
MetricValTest
Depth AbsRel ↓0.1480.163
Depth δ1 ↑0.8310.790
Rotation AUC @5 / 15 / 300.178 / 0.480 / 0.6710.176 / 0.461 / 0.663
Translation AUC @5 / 15 / 300.009 / 0.056 / 0.1560.003 / 0.034 / 0.120
Joint Pose AUC @5 / 15 / 300.005 / 0.043 / 0.1360.001 / 0.023 / 0.108

GT vs. prediction (left: GT, right: prediction)

09c1414f1b GT
09c1414f1b · GT
09c1414f1b prediction
09c1414f1b · prediction
21d970d8de GT
21d970d8de · GT
21d970d8de prediction
21d970d8de · prediction
280b83fcf3 GT
280b83fcf3 · GT
280b83fcf3 prediction
280b83fcf3 · prediction
2b1dc6d6a5 GT
2b1dc6d6a5 · GT
2b1dc6d6a5 prediction
2b1dc6d6a5 · prediction

Errors on held-out scenes

Depth is the best part: the room layout is mostly right, but surfaces are noisier and flat walls get rounded (e.g. 280b83fcf3). Translation direction is the weakest metric. In 2b1dc6d6a5 the GT cameras are spread along a path, while the predicted ones are packed into two small clusters. The 8 views come from only 32 frames, so baselines are short and hard to estimate. The camera loss was also still dropping when training stopped. Test is a bit worse than val, mainly on δ1 and translation.

3.3 Bonus: Number of Views

Bonus · 10 points

Question. The model was trained with 8 views. What happens to accuracy, runtime and memory if I give it a different number of views at test time?

Setup. Same epoch-721 checkpoint, no retraining. Validation split, first 32 frames, seed 825. Only the number of views changes (2, 4, 8, 16, 24, 32). Time and memory are measured with batch size 1.

Views ablation
Dashed line: training setting (8 views)
ViewsAbsRel ↓δ1 ↑Rot AUC@30Trans AUC@30Pose AUC@30Time (ms)Memory (GB)
20.1170.8600.6440.1950.162480.15
40.1380.8340.6600.1830.157590.17
80.1480.8310.6710.1560.136720.22
160.1720.7980.6270.1370.113730.30
240.1750.7770.6380.1350.113750.38
320.1830.7710.6330.1330.1131110.47

Results. More views did not help. Depth and translation get steadily worse as views are added. Rotation is best at 8 views, the same number used in training. Memory grows about linearly. Runtime is flat from 8 to 24 views and jumps at 32.

Why. I think the main reason is the train/test mismatch: global attention only ever saw 8 views during training. The task also changes with the number of views. With more views, the set covers more of the 32-frame window and the scene scale changes, so the rows are not exactly the same problem.

Limitations. I only changed the number of views at test time. With 2 views there is just one pair per scene, so those pose numbers are noisy, and the timings come from a shared GPU. Training with a random number of views would be the natural next step.