16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Unprojection (original world frame, meters, z up)

Canonicalization (camera 0 center and centroid distance are in the right panel)

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.777836385183036e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011736395390471444,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanation

2. Novel-View Synthesis with Mini-LVSM

2.1

Final overfit PSNR (2000 steps, one fixed batch): 30.99 dB (SSIM 0.8710).


val_epoch000.png — columns: source views, ground truth, prediction

2.2

Training settings: Default configs/lvsm.json, except 600 epochs instead of 100. With 100 epochs, val PSNR was only 16.14 dB and still rising. Batch 4, 8 views per sample, 224 px, S = 2 source views (6 targets). Patch 8, width 384, 8 layers. AdamW, lr 2e-4, weight decay 0.05, 200 warm-up steps, cosine decay, bf16. One GPU. best.pt is from epoch 595.


Training curves

Validation metrics (best.pt, 28 val scenes):

Source viewsTarget raysPSNR ↑SSIM ↑
1correct16.060.5961
1shuffled11.590.4779
1none12.320.5193
2correct17.480.6107
2shuffled11.160.4555
2none12.290.5169
4correct18.690.6351
4shuffled11.200.4522
4none12.350.5285


Training visualization (val_epoch595.png) — columns: source views, ground truth, prediction

Evaluation visualizations (columns: source views, ground truth, prediction):


S=1, correct rays


S=2, correct rays


S=4, correct rays


S=2, correct rays


S=2, shuffled rays


S=2, none rays

Discussion:

2.3 Bonus: training with a random number of source views

Question: if we train with a random number of source views, does the model handle 1, 2 and 4 views better?

Setting: same as Q2.2 (same config, 600 epochs, same seed). The only change: each training step picks S at random from {1, 2, 3, 4} (option train_source_views, configs/lvsm_randS.json). Validation and best.pt selection still use S = 2.

Eval STarget raysPSNR fixed S=2PSNR random SΔ PSNRSSIM fixed S=2SSIM random S
1correct16.0616.10+0.030.59610.5978
1shuffled11.5911.62+0.030.47790.4810
1none12.3212.54+0.220.51930.5298
2correct17.4817.60+0.120.61070.6137
2shuffled11.1611.15-0.010.45550.4536
2none12.2912.51+0.220.51690.5258
4correct18.6918.96+0.260.63510.6408
4shuffled11.2011.13-0.060.45220.4435
4none12.3512.53+0.180.52850.5339


Training curves (validation uses S=2)


fixed S=2 model, eval S=1


random-S model, eval S=1


fixed S=2 model, eval S=4


random-S model, eval S=4

Conclusions:

3. Multi-View Geometry with Mini-VGGT

3.1


Overfit loss curve (total, camera, depth)

Ground truthPrediction (predicted camera + depth)

3.2

Training settings: Default configs/vggt.json: 1000 epochs, 8 views per sample, 224 px, batch 16 on 2 GPUs (DDP). Frozen DINOv2 ViT-B/14, 8 frame/global blocks (width 384). AdamW, lr 6e-4, weight decay 0.01, 10 warm-up epochs, cosine decay. Loss weights: camera 5, depth 1. bf16. best.pt (lowest val loss) is from epoch 952.


Normal-training loss curve (total, camera, depth; train and validation)

Metrics (best.pt, 28 scenes per split, 8 views):

SplitAbsRel ↓δ1 ↑Rot AUC@5Rot AUC@15Rot AUC@30Trans AUC@5Trans AUC@15Trans AUC@30Pose AUC@5Pose AUC@15Pose AUC@30
val0.13750.84310.16890.49490.69300.01380.08300.20270.00480.06660.1806
test0.15510.79180.20840.51510.70970.01020.06390.19100.00100.03460.1628

GT vs. prediction, four val scenes:

Ground truthPrediction

Discussion:

3.3 Bonus: number of input views at evaluation

Question: the model is trained with 8 views. What happens to accuracy, runtime and memory with 2, 4, 8, 16 or 24 views at test time?

Setting: same Q3.2 checkpoint, no retraining (scripts/vggt_num_views.py). Views come from the first 32 frames, seed 825. Runtime: average of 10 forward passes on 4 samples, divided by 4. Memory: peak GPU memory during the forward pass (without weights).

val

ViewsAbsRel ↓δ1 ↑Rot AUC@30Trans AUC@30Pose AUC@5Pose AUC@30Forward ms / samplePeak inference memory (MB)
20.12050.85450.70000.26670.01430.22505.2400
40.12350.85840.67020.21110.00360.18554.4450
80.13750.84300.69350.20350.00510.18148.1769
160.16660.81000.65930.16700.00180.141217.21411
240.16970.79600.66720.17260.00200.148325.42054

test

ViewsAbsRel ↓δ1 ↑Rot AUC@30Trans AUC@30Pose AUC@5Pose AUC@30Forward ms / samplePeak inference memory (MB)
20.12070.83890.72500.19050.00000.15245.6400
40.14070.81530.70750.22620.00360.18354.8450
80.15510.79170.71020.19080.00100.16278.0769
160.16980.77990.70780.18480.00210.158917.21411
240.16870.77710.72090.18650.00180.162125.52054


Accuracy, runtime and memory vs. number of views

Conclusions: