Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision · Andrew ID: jfu2

1. Multi-View Data Preprocessing

1.2 Unprojection (original world frame, meters)

Unprojection screenshot
Points and cameras after Q1.1–1.2, in the original world frame.

1.3 Canonicalization

Canonicalization screenshot
Canonicalized geometry. Camera 0 is the identity at the origin; the valid-point centroid is at distance 1.

1.7 check_data.py output

SampleViewsRes.Valid depthMax point reproj. (px)Max ray reproj. (px)Centroid dist.
e050c15a8d_frame_00708082240.9781.13e-41.09e-41.0000
5f99900f09_frame_00282082240.9901.23e-41.11e-41.0000
47b37eb6f9_frame_00272082240.9619.88e-58.90e-51.0000
4c5c60fa76_frame_00669082240.7521.16e-41.07e-41.0000

Batch shapes (B=2, V=8): rgb (2,8,3,224,224), depth (2,8,1,224,224), K (2,8,3,3), T_world_from_camera (2,8,4,4), points_world (2,8,3,224,224), ray_origins/ray_directions (2,8,3,224,224), plucker (2,8,6,224,224), scene_scale (2). Reprojection errors are about 1e-4 px (threshold 0.01 px) and the centroid distance is 1.

Explanations

Canonicalization. Every training sample is re-expressed in the coordinate frame of its first camera: each pose becomes T_i' = T_0-1 T_i, so camera 0 is the identity. The scene is then divided by one shared scale, the distance from the origin to the centroid of all valid points, applied to translations, depth and points (rotations are unchanged and the centroid is not subtracted). This removes the arbitrary world origin, orientation and metric unit of each scan, so every sample has the same conventions and a similar size, and the model only has to predict geometry relative to the first view.

World-space ray directions. For pixel (u,v) the camera-space direction is K-1[u,v,1]T (the point at z = 1). Rotating it by the camera-to-world rotation R and normalizing gives a unit direction in the canonical world frame; the ray origin is the camera center (the pose translation). Unlike unprojection, no depth is used, so a ray describes only where a pixel looks, which is why it is available for a target view whose depth is unknown.

Plücker moment. A ray is encoded as [d, m] with m = o × d. Because d is a unit vector, |m| is the distance from the world origin to the line, and m is the same for any point o chosen on the ray (moving o by t d adds t (d × d) = 0). The six numbers therefore identify a 3D line uniquely and vary smoothly, which makes them a convenient per-pixel camera encoding.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit resultPSNR (dB)SSIM
Validation on the fixed batch30.980.872

The training PSNR of 25.40 dB is the average over all 2000 steps, which includes the early steps from random initialization.

Overfit visualization
val_epoch000.png: source views, ground truth, prediction (left to right).

2.2 Training and evaluation

Training settings. Mini-LVSM, 14.6M parameters: patch size 8, width 384, 8 transformer layers, head dim 64, MLP ratio 4, QK-norm. 8 views per sample at 224×224, of which 2 are sources and 6 are targets; batch size 4 scenes, 56 steps per epoch, 100 epochs (5,600 steps). AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps then cosine decay to 10%, gradient clipping 1.0, bfloat16 autocast, MSE loss, seed 825. About 28 s per epoch (≈47 min).

Ablation: number of source views × target-ray conditioning (validation)

Source viewsConditioningPSNR (dB)SSIM#target images
1correct15.200.558196
1shuffled11.790.487196
1none12.460.530196
2correct16.150.560168
2shuffled11.480.472168
2none12.430.528168
4correct16.900.575112
4shuffled11.580.472112
4none12.460.538112

correct: true target rays; shuffled: rays of a different target view; none: all-zero rays. With a fixed 8 views per sample, more source views leave fewer target images, so the three source settings are averaged over different target sets.

Visualizations

Training visualizations on validation data. Each row is one target view; columns are the source views, the ground truth, then the prediction.

epoch 0
Epoch 0
epoch 50
Epoch 50
epoch 95
Epoch 95

Evaluation visualizations of the best checkpoint (rows: targets; columns: sources, ground truth, prediction).

S1 correct
1 source, correct
S1 shuffled
1 source, shuffled
S1 none
1 source, none
S2 correct
2 sources, correct
S2 shuffled
2 sources, shuffled
S2 none
2 sources, none
S4 correct
4 sources, correct
S4 shuffled
4 sources, shuffled
S4 none
4 sources, none

Discussion

2.3 Bonus: Analysis and Exploration

Question. Does adding an SSIM term to the MSE loss improve SSIM, PSNR and visual sharpness of the synthesized views?

Setup. Same model, data, schedule and seed as the Q2.2 baseline; only the loss changes from MSE to MSE + 0.1 · (1 − SSIM) (SSIM averaged over all target images, enabled with LVSM_LOSS=mse_ssim). Both runs are evaluated on the validation split (28 scenes) at the checkpoint with the highest validation PSNR (the selection rule of the training script), with the same source-view and ray-conditioning ablation as in Q2.2. The first table is the training configuration (2 source views, correct rays). Each setting was trained once.

LossVal PSNR (dB)Val SSIMBest epochSeconds / epoch
MSE (baseline)16.150.5609927.9
MSE + 0.1 · (1 − SSIM)15.530.5779628.4
Difference−0.62+0.017+0.5

Epochs are counted from 0. Because the best checkpoint is chosen by validation PSNR, the PSNR of the SSIM run is not penalized by the selection; the validation loss of the two runs is not comparable because it includes the SSIM term in one run.

Source viewsConditioningMSE PSNRMSE SSIMMSE+SSIM PSNRMSE+SSIM SSIMΔ PSNRΔ SSIM
1correct15.200.55814.870.575−0.34+0.017
1shuffled11.790.48711.870.513+0.07+0.027
1none12.460.53012.410.544−0.05+0.014
2correct16.150.56015.530.577−0.62+0.017
2shuffled11.480.47211.660.506+0.18+0.034
2none12.430.52812.390.540−0.05+0.012
4correct16.900.57515.940.591−0.95+0.016
4shuffled11.580.47211.810.513+0.23+0.041
4none12.460.53812.420.550−0.03+0.011
MSE epoch 95
MSE loss, epoch 95 (columns: two source views, ground truth, prediction).
MSE+SSIM epoch 95
MSE + SSIM loss, epoch 95 (same layout).

Evaluation visualizations of the best checkpoints (2 source views, correct rays; rows: targets; columns: two sources, ground truth, prediction).

MSE, 2 sources, correct
MSE loss
MSE+SSIM, 2 sources, correct
MSE + SSIM loss

Discussion

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

VGGT overfit loss curve
Overfit loss (total, camera, depth).
Epoch 40, fixed clipDepth AbsRelDepth δ1Rot. errorTrans. errorPose AUC@5Pose AUC@15
Overfit0.0360.9560.26°0.54°0.9570.983
Overfit GT vs prediction
Ground truth (left) and prediction (right) for the overfit clip. The cameras and the room layout match closely; the predicted point cloud is slightly noisier.

3.2 Training and evaluation

Training settings. Frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (16 blocks), width 384, 6 heads; camera head with cross-view blocks and a depth head with RGB refinement. 8 views per sample at 224×224, sampling range 32, batch size 16 (14 steps per epoch), 1000 epochs (≈14k steps). AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs then cosine decay to 5%, loss weights camera 5.0 and depth 1.0, bfloat16, 4 data workers, seed 825. About 21 s per epoch (≈6 h on one A10G). best.pt is the checkpoint with the lowest validation loss.

VGGT training curves
Training and validation loss (total, camera, depth).

Evaluation: 8 views from the first 32 prepared frames, 224 px, seed 825, full splits (28 scenes each).

MetricValidationTest
Depth AbsRel (lower is better)0.1490.164
Depth δ10.8150.793
Rotation AUC @5° / 15° / 30°0.189 / 0.499 / 0.6940.203 / 0.523 / 0.720
Translation AUC @5° / 15° / 30°0.017 / 0.081 / 0.1910.007 / 0.060 / 0.172
Joint pose AUC @5° / 15° / 30°0.001 / 0.059 / 0.1670.001 / 0.041 / 0.155
Mean rotation error12.2°9.8°
Mean translation-direction error41.8°41.0°

GT vs. predicted geometry (four validation scenes)

Left: ground truth; right: predicted cameras and depth (frozen-DINO baseline, best.pt). Camera 0 is red. The two panels were captured in the interactive viewer and their viewing angles are not identical.

Scene 1
Scene 1 (09c1414f1b, dining room): the camera cluster and the overall room outline are recovered, but the predicted point cloud is bloated and the window and wall structure is lost.
Scene 2
Scene 2 (21d970d8de): the cameras are placed close to the ground truth, but the predicted points are noisier and the walls and ceiling are stretched and scattered.
Scene 3
Scene 3 (280b83fcf3, narrow hallway): the cameras are close to the ground truth, but the long thin hallway is compressed and deformed in the prediction.
Scene 4
Scene 4 (2b1dc6d6a5, office): the camera order along the trajectory is preserved but their spacing differs, and the point cloud is smeared diagonally so the chair is barely recognizable.

Discussion

3.3 Bonus: Analysis and Exploration

Question. How does the number of input views affect Mini-VGGT's accuracy and cost at evaluation time?

Setup. We evaluate the single baseline checkpoint from Q3.2 (best.pt, trained with 8 views and a frozen DINOv2) on the full test split (28 scenes) with --num-views set to 2, 4, 8 and 16, and nothing else changed (batch size 4, seed 825, 224 px, one NVIDIA A10G). Views are the reference frame 0 plus the first V−1 frames of a per-scene seeded random permutation of the first 32 prepared frames, so the sets are nested (the 2-view set is contained in the 4-view set, and so on). The 8-view row matches the Q3.2 test result up to small numerical differences (at most about 0.002 in the AUC values; this sweep used batch size 4). No model was retrained.

ViewsPose pairsDepth AbsRel ↓Depth δ1 ↑Depth RMSE (norm.) ↓Eval time (s)
2280.1300.8490.1916
41680.1500.8200.1977
87840.1640.7930.2078
1633600.1750.7690.23013
ViewsRot. errorTrans. errorRotation AUC @5°/15°/30°Translation AUC @5°/15°/30°Joint AUC @5°/15°/30°
210.2°44.2°0.279 / 0.545 / 0.7290.007 / 0.110 / 0.2290.000 / 0.052 / 0.161
48.9°38.2°0.236 / 0.576 / 0.7570.006 / 0.071 / 0.1920.005 / 0.050 / 0.165
89.8°41.0°0.202 / 0.523 / 0.7190.007 / 0.060 / 0.1720.002 / 0.041 / 0.154
169.7°41.5°0.192 / 0.523 / 0.7200.010 / 0.067 / 0.1800.001 / 0.043 / 0.156

Eval time is the wall-clock time of the whole evaluation command (Python start-up, checkpoint loading, data loading and the forward passes), so it is only a coarse indication of cost; GPU memory was not measured. Pose metrics use all pairs of views, so the number of pairs grows as V(V−1)/2 per scene.

Discussion