Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

1.1 Resizing camera intrinsics

s = 224 / 300 f' = s · f c' = s · (c + 0.5) − 0.5

The +0.5 is there because pixel centers are at integer coordinates, so the image edge is at −0.5.

1.2 Depth unprojection

X_world = R · ( depth(u, v) · K⁻¹ [u, v, 1]ᵀ ) + t

Points and cameras in the original world frame (meters). Red dot: camera 0. Blue dots: the other cameras.

1.3 Camera and depth canonicalization

T_i ← (T_0)⁻¹ · T_i s = ‖centroid of valid points‖ t, depth, points ← ÷ s

The same scenes after canonicalization. Each side panel shows camera 0's center, (0, 0, 0), and a valid-point centroid distance of 1.000.

1.4 Camera rays

o = t d = normalize( R · K⁻¹ [u, v, 1]ᵀ )

1.5 Plücker coordinates

plucker = [ d, m ], m = o × d

1.7 Final check

SampleValid depthPoint reproj. error (px)Ray reproj. error (px)Centroid distance
e050c15a8d_frame_00708097.8%0.0001130.0001091.000
5f99900f09_frame_00282099.0%0.0001230.0001111.000
47b37eb6f9_frame_00272096.1%0.0000990.0000891.000
4c5c60fa76_frame_00669075.2%0.0001160.0001071.000

All reprojection errors are far below the 0.01 px limit, and the shapes match the data contract.

Full check_data.py output
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

2. Novel-View Synthesis with Mini-LVSM

In every visualization, the columns are: source views, ground truth, prediction.

2.1 Overfitting

After 2,000 steps on one fixed batch, the final PSNR is 30.87 dB (SSIM 0.869).

2.2 Training and evaluation

Training settings: configs/lvsm.json defaults. Patch size 8, width 384, 8 layers, head dim 64 (14.6M parameters). 2 source and 6 target views out of 8, at 224×224. Batch 4 for 100 epochs (5,600 steps). AdamW with lr 2×10−4, weight decay 0.05, 200 warmup steps then cosine decay, gradient clip 1.0, bf16. Training took 59 min on one NVIDIA L4. The best checkpoint is from epoch 99.

Source viewsCorrect rays: PSNR / SSIMShuffled rays: PSNR / SSIMNo rays: PSNR / SSIM
115.18 / 0.55811.80 / 0.48812.47 / 0.530
216.13 / 0.56011.50 / 0.47312.44 / 0.527
416.87 / 0.57511.60 / 0.47412.46 / 0.537

Validation split, best.pt.

2.3 Bonus: training view count

Question. The baseline always trains with 2 source views. Does training with a random number of source views per step make one model better at 1, 2 and 4 views?

Setup. Same config, seed and schedule as 2.2. The only change is that every batch draws S uniformly from {1, 2, 3, 4} (configs/lvsm_random_views.json). Validation during training uses S = 2 for both runs. Both best.pt checkpoints are evaluated with correct rays on the validation split. The random-S run took 61 min.

Source views at testFixed S = 2: PSNR / SSIMRandom S: PSNR / SSIMΔ PSNR
115.18 / 0.55815.27 / 0.559+0.09
216.13 / 0.56016.22 / 0.560+0.09
416.87 / 0.57517.09 / 0.575+0.22

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

One fixed training clip (8 views of 036bce3393), 40 epochs × 25 steps. At the end: depth AbsRel 0.037, δ1 0.955, mean rotation error 0.22°, translation-direction error 0.43°, joint pose AUC@30 0.996.

3.2 Training and evaluation

Training settings: configs/vggt.json defaults. Frozen DINOv2 ViT-B/14, width 384, 6 heads, 8 frame/global pairs (57.7M trainable parameters). 8 views, sampling range 32, 224×224, batch 16. 1,000 epochs (14,000 steps) with AdamW, lr 6×10−4, weight decay 0.01, 10 warmup epochs then cosine decay to 5%, gradient clip 1.0, bf16. Loss = 5 × camera + 1 × depth. Training took 9.1 h on one NVIDIA L4. best.pt is from epoch 929 (val loss 0.872).

SplitDepth AbsRel ↓Depth δ1 ↑Rotation AUC @5 / 15 / 30Translation AUC @5 / 15 / 30Joint pose AUC @5 / 15 / 30
Val0.1460.8260.192 / 0.504 / 0.6920.011 / 0.064 / 0.1710.002 / 0.049 / 0.152
Test0.1600.8010.206 / 0.525 / 0.7180.014 / 0.065 / 0.1580.002 / 0.042 / 0.140

best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825. 28 scenes and 784 view pairs per split.

3.3 Bonus: number of input views

Question. How does the number of input views change accuracy, runtime and memory of the model trained with 8 views?

Setup. The same best.pt on the val split with 2, 4, 8 and 16 views from the first 32 prepared frames. Batch 4, 224 px, seed 825, no retraining. The loader always takes frame 0 plus a seeded random prefix of the other frames, so the view sets are nested (2 ⊂ 4 ⊂ 8 ⊂ 16). That makes a controlled comparison possible: besides scoring all views, I also score only the 2 views every run shares, so only the extra context views change. Time is the model's forward pass per sample. Memory is the peak allocated GPU memory, which includes 0.58 GB of weights (144M parameters, fp32).

ViewsAll views: AbsRel / δ1 / joint AUC@30Shared 2 views: AbsRel / δ1 / rot. AUC@30 / trans. AUC@30 / joint AUC@30Time per sample (ms)Peak memory (GB)
20.130 / 0.840 / 0.2540.130 / 0.840 / 0.701 / 0.277 / 0.2548.30.80
40.142 / 0.835 / 0.1700.134 / 0.848 / 0.706 / 0.208 / 0.19813.30.88
8 (training)0.146 / 0.826 / 0.1520.138 / 0.850 / 0.710 / 0.213 / 0.19526.91.02
160.173 / 0.779 / 0.1320.166 / 0.809 / 0.704 / 0.231 / 0.20560.31.32