Assignment 4: Multi-View Transformers

Author: Po-Ting Ko

Andrew ID: potingk

1. Multi-View Data Preprocessing

1.1 Resizing camera intrinsics

s = Sout / Sin,    f′ = s f,    c′ = s (c + 0.5) − 0.5

The +0.5 is there because pixel centers sit at integer coordinates, so the image edge is at −0.5.

1.2 Depth unprojection

Xworld = R ( z(u, v) K−1 [u, v, 1]T ) + t

Points in the original world frame (meters). The red frustum is the first sampled camera.

Scene 036bce3393 Scene 07f5b601ee
Unprojection scene 0 Unprojection scene 3

1.3 Camera and depth canonicalization

Ti ← T0−1 Ti,    s = ‖centroid of valid points‖,    t, depth, points ← ÷ s

Why canonicalize? A scan's world frame (origin, orientation, size) is arbitrary and cannot be predicted from images. Working relative to view 0 removes the rigid-transform ambiguity, and fixing the centroid distance to 1 removes the scale ambiguity.

The viewer's side panel, enlarged under each screenshot, shows the first camera center and the valid-point centroid distance. All 5 scenes give (0, 0, 0) and 1.000, with scene scales from 0.924 to 2.830 m.

Scene 036bce3393 (s = 1.867 m) Scene 07f5b601ee (s = 0.924 m)
Canonicalization scene 0 Canonicalization scene 3
Scene 0 first camera center and centroid distance Scene 3 first camera center and centroid distance

1.4 Camera rays

o = t,    d = normalize( R K−1 [u, v, 1]T )

World-space directions. Camera-space directions depend only on K, so they are the same for every view. In the shared frame, rays of different views become comparable: rays of pixels that see the same 3D point intersect there.

1.5 Plücker coordinates

plücker = [d, m],    m = o × d

Plücker moment. Every point o + λd on the line gives the same moment. It is perpendicular to the plane through the line and the origin, and its norm is the line's distance from the origin. So (d, m) identifies the line, and each pixel encodes both camera orientation and position.

1.7 Final check

Sample Valid depth Point reproj. error (px) Ray reproj. error (px) Centroid distance
e050c15a8d_frame_007080 0.978 1.11e-4 1.08e-4 1.000
5f99900f09_frame_002820 0.990 1.23e-4 1.11e-4 1.000
47b37eb6f9_frame_002720 0.961 9.78e-5 8.90e-5 1.000
4c5c60fa76_frame_006690 0.752 1.12e-4 1.02e-4 1.000

All reprojection errors are far below the 0.01 px limit, and the shapes match the data contract.

Full check_data.py output
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011141905270051211,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.777836385183036e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011162471491843462,
      "max_ray_reprojection_error_px": 0.00010225349979009479,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

In every visualization, the columns are: source views, ground truth, prediction.

2.1 Overfitting

After 2,000 steps on one fixed batch, the final PSNR is 34.02 dB (SSIM 0.915).

LVSM overfit

2.2 Training and evaluation

Training settings: configs/lvsm_local_300ep.json. Same as configs/lvsm.json except batch 2 (batch 4 exceeds 8 GB) and 300 epochs (33,600 steps), because the default 100 epochs reached 16.16 dB and was still improving. Patch size 8, width 384, 8 layers, head dim 64, QK-norm (14.6M parameters). 2 source and 6 target views out of 8, sampling range 32. AdamW with lr 2e-4, betas (0.9, 0.95), weight decay 0.05, 200 warmup steps then cosine decay, gradient clip 1.0, bf16. Training took 3 h 40 min on an RTX 5060 Laptop GPU (8 GB). best.pt is from epoch 296.

LVSM training curves

Source views Correct rays: PSNR / SSIM Shuffled rays: PSNR / SSIM No rays: PSNR / SSIM
1 15.69 / 0.580 11.67 / 0.483 12.49 / 0.538
2 16.86 / 0.586 11.26 / 0.462 12.45 / 0.534
4 17.89 / 0.604 11.30 / 0.461 12.48 / 0.544

Validation split, best.pt. As a baseline, copying the most similar source view gives only 11.9 dB (S = 2).

Training: epoch 0 Training: epoch 295
Epoch 0 Epoch 295
Evaluation: S = 1, correct rays Evaluation: S = 4, correct rays
S=1 S=4
S = 2, correct rays S = 2, shuffled rays S = 2, no rays
Correct Shuffled None

2.3 Bonus: adding an SSIM loss

Question. MSE averages over plausible images, which causes blur. Does an SSIM term give sharper novel views, and at what cost in PSNR?

Setup. The loss is MSE + λ (1 − SSIM), with λ = 0.05 and 0.2. On the trained model the SSIM gradient is about 7× the MSE gradient, so λ ≈ 0.14 balances them. Everything else matches a 100-epoch MSE control run. scripts/lvsm_sharpness.py measures two ratios on 168 validation targets: sharpness (mean image gradient of the prediction over that of GT) and blockiness (mean pixel difference across 8-px patch borders over that inside patches; GT gives 1.01).

Q2.3 validation curves

Loss (100 epochs) S = 1: PSNR / SSIM S = 2: PSNR / SSIM S = 4: PSNR / SSIM Sharpness ↑ Blockiness ↓
MSE (control) 15.20 / 0.559 16.16 / 0.561 16.86 / 0.576 0.299 7.79
MSE + 0.05 (1 − SSIM) 14.49 / 0.567 15.01 / 0.569 15.23 / 0.581 0.153 4.07
MSE + 0.2 (1 − SSIM) 14.48 / 0.571 14.97 / 0.573 15.13 / 0.584 0.131 3.20
MSE, 300 epochs (Q2.2) 15.69 / 0.580 16.86 / 0.586 17.89 / 0.604 0.328 5.47

Q2.3 visual comparison

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

One fixed clip, 40 epochs × 25 steps. At the end: rotation error 0.22° (from 49.7°), translation-direction error 0.35° (from 90°), depth AbsRel 0.037 (from 0.619). Total and depth losses go negative because of the −0.2 log c confidence term.

VGGT overfit loss

Ground truth Prediction
Overfit GT Overfit prediction

3.2 Training and evaluation

Training settings: configs/vggt_local.json. Same as configs/vggt.json except batch 4 × 4 gradient-accumulation steps, so each update still sees 16 samples (batch 16 does not fit in 8 GB; I added training.grad_accum_steps to models/vggt/train.py). Frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads (57.7M trainable parameters). 8 views within 32 frames, 224 px. 1,000 epochs with AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs then cosine decay to 5%, gradient clip 1.0, bf16. Training took 5.9 h on an RTX 5060 Laptop GPU (8 GB). best.pt is from epoch 797 (val loss 0.864).

VGGT training loss

Train loss is above validation loss because training samples 32-frame windows anywhere in a sequence, while validation always uses the first 32 frames.

Split AbsRel ↓ δ1 ↑ Rotation AUC @5 / 15 / 30 Translation AUC @5 / 15 / 30 Joint pose AUC @5 / 15 / 30
Val 0.140 0.846 0.177 / 0.479 / 0.679 0.018 / 0.076 / 0.175 0.005 / 0.055 / 0.154
Test 0.173 0.780 0.173 / 0.489 / 0.692 0.006 / 0.050 / 0.142 0.002 / 0.035 / 0.126

best.pt, 8 views from the first 32 frames, seed 825. 28 scenes and 784 view pairs per split.

Comparison with the TA reference. I ran my model on the same 8-view samples as the 4 scenes in the TA viewer (GT points match exactly) and computed the same metrics for both from the viewer data (scripts/compare_ta_reference.py):

4 TA scenes, mean Rotation error ↓ Translation-direction error ↓ AbsRel ↓ δ1 ↑ Point L2 ↓
TA reference 33.7° 46.8° 0.182 0.778 0.530
Mine 28.4° 40.5° 0.148 0.841 0.449

The gain comes mainly from be0ed6b33c and fd361ab85f. On 394a542a19 and 7b6477cb95 the two models are close, and each wins some metrics. With 4 scenes this is a sanity check, not a ranking.

Scene Ground truth Prediction
09c1414f1b 09c1414f1b GT 09c1414f1b prediction
21d970d8de 21d970d8de GT 21d970d8de prediction
280b83fcf3 280b83fcf3 GT 280b83fcf3 prediction
2b1dc6d6a5 2b1dc6d6a5 GT 2b1dc6d6a5 prediction

3.3 Bonus: number of input views

Question. The model is trained with 8 views. How do accuracy, runtime and memory change with V at test time?

Setup. The same best.pt with V ∈ {2, 4, 8, 16, 24, 32} from the first 32 frames, seed 825, no retraining. View sets are nested, so core metrics score only the 4 views every run shares, and only the extra context changes. Aligned AbsRel first scales each sample's depth by median(GT / prediction), separating depth shape from global scale. Runtime and memory use fixed inputs (batch 1, median of 20 runs) for V up to 128. One more sweep draws 8 views from the first 128 frames.

Number of views study

V All: AbsRel / aligned / pose AUC@30 Core: AbsRel / aligned / pose AUC@30 Time (ms) Memory (GiB)
2 0.110 / 0.081 / 0.163 – 27.0 0.70
4 0.131 / 0.086 / 0.132 0.131 / 0.086 / 0.132 30.8 0.71
8 (training) 0.140 / 0.089 / 0.154 0.139 / 0.085 / 0.147 45.8 0.75
16 0.170 / 0.094 / 0.129 0.171 / 0.086 / 0.142 83.2 0.84
32 0.179 / 0.095 / 0.124 0.181 / 0.086 / 0.142 172.4 0.98
64 – – 405.1 1.29
128 – – 1054.2 1.89

Validation split. Test shows the same trends, with core aligned AbsRel 0.095–0.096 for every V.

V = 2 V = 8 V = 32
V=2 V=8 V=32