Multi-View Transformers


1. Multi-View Data Preprocessing

Unprojection

unprojection

Canonicalization

canonicalization

check_data.py

scene valid depth max reproj err (px) max ray reproj err (px) centroid dist
e050c15a8d_frame_007080 0.978 1.13e-4 1.09e-4 1.000
5f99900f09_frame_002820 0.990 1.23e-4 1.11e-4 1.000
47b37eb6f9_frame_002720 0.961 9.88e-5 8.90e-5 1.000
4c5c60fa76_frame_006690 0.752 1.16e-4 1.07e-4 1.000

Canonicalization: We express camera poses as related to camera 0, then we normalize translations, depth and points by dividing them by the distance from camera 0 to the centroid of valid points.

World-space ray directions: d = normalize(R K⁻¹ [u, v, 1]ᵀ), the pixel's back-projected direction rotated into the canonical world frame, and the origin is the camera center.

Plücker moment: m = o × d. It encodes both direction and position of the ray.


2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 30.86 dB (SSIM 0.869), 2000 steps.

overfit

2.2 Training and Evaluation

Settings: patch 8, width 384, 8 layers, head dim 64, 2 source / 6 target views, batch 4, 100 epochs, AdamW lr 2e-4, wd 0.05, warmup 200, bf16.

curves

Training visualization (epoch 95):

train vis

source views conditioning PSNR SSIM
1 correct 15.19 0.558
1 shuffled 11.81 0.487
1 none 12.46 0.530
2 correct 16.13 0.560
2 shuffled 11.48 0.472
2 none 12.43 0.528
4 correct 16.87 0.575
4 shuffled 11.60 0.473
4 none 12.44 0.538
S=1, correct S=2, correct S=4, correct
S=2, shuffled S=2, none

Source views: More source views raise PSNR since more of the target is observed, even when the model was trained with 2 views only.

Ray conditioning: Without target rays the model outputs a nearly flat mean color. With shuffled rays it renders structure from the wrong viewpoint, which is worse than no rays in PSNR. This shows that the model is conditioned on target rays to choose the view point.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

overfit loss

GT (left) vs. prediction (right):

overfit gt pred

3.2 Training and Evaluation

Settings: frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads, 8 views, sampling range 32, batch 16, 1000 epochs, lr 6e-4, wd 0.01, warmup 10, loss weights camera 5 / depth 1

loss curve

split AbsRel ↓ δ1 ↑ Rot AUC 5/15/30 ↑ Trans AUC 5/15/30 ↑ Pose AUC 5/15/30 ↑
val 0.159 0.793 0.174 / 0.505 / 0.694 0.012 / 0.087 / 0.192 0.003 / 0.068 / 0.170
test 0.171 0.765 0.205 / 0.545 / 0.722 0.011 / 0.061 / 0.170 0.003 / 0.046 / 0.153

GT (left) vs. prediction (right):

scene 0 scene 1 scene 2 scene 3

Errors on held-out scenes: Depth, rotation, and the room's layout generalizes well, but point clouds are smoother, and reconstructions looks bad on thin structures. Since the input cameras are close to each other, so small position errors give large angle errors and poor translation/pose AUC. But camera loss was still dropping at 1000 epochs, so longer training might help improve the performance.