16-825 Assignment 4: Multi-View Transformers

Abhikhya Tripathy (Andrew ID: abhikhyt)

Part 1: Data

Unprojection

Depth maps from 8 sampled views unprojected into world coordinates (blue: cameras, red: first camera).

Unprojection, scene 036bce3393 Unprojection, scene 076c822ecc

Canonicalization

Points and cameras expressed in the first camera's frame and rescaled. The side panel reports the first camera center (0, 0, 0) and the valid-point centroid distance (≈ 1).

Canonicalization, scene 036bce3393
036bce3393: first camera center (0.0000, 0.0000, 0.0000), valid-point centroid distance 1.000 (scene scale 1.867 m)
Canonicalization, scene 076c822ecc
076c822ecc: first camera center (0.0000, 0.0000, 0.0000), valid-point centroid distance 1.000 (scene scale 2.830 m)
Canonicalization, scene 079a326597
079a326597: first camera center (0.0000, 0.0000, 0.0000), valid-point centroid distance 1.000 (scene scale 2.080 m)
Canonicalization, scene 07f5b601ee
07f5b601ee: first camera center (0.0000, 0.0000, 0.0000), valid-point centroid distance 1.000 (scene scale 0.924 m)

check_data.py output

Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4. The 4 checked samples (out of 224 train samples):

SampleViewsResolutionValid depth fractionMax point reprojection error (px)Max ray reprojection error (px)Centroid distance
e050c15a8d_frame_00708082240.9781.11e-041.07e-041.000
5f99900f09_frame_00282082240.9901.23e-041.22e-041.000
47b37eb6f9_frame_00272082240.9611.02e-048.90e-051.000
4c5c60fa76_frame_00669082240.7521.09e-041.03e-041.000

All reprojection errors are about 1e-4 px, well below the 0.01 px requirement, and every centroid distance is 1.0. Full output (a PyTorch DataLoader worker-count warning printed before the JSON is omitted):

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011141905270051211,
      "max_ray_reprojection_error_px": 0.00010682854917831719,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001220703125,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. All cameras and points are moved into the first camera's frame, so camera 0 sits at the origin with no rotation. Everything is then divided by one scale factor, the distance from the origin to the average of the valid 3D points, so that this average ends up at distance 1. This way every sample uses the same coordinates and the same scale, whatever the original world frame and room size.

World-space ray directions. For each pixel, K⁻¹ · (u, v, 1) gives the direction it looks in, in camera coordinates. Rotating it by the camera's rotation turns it into world coordinates, and it is scaled to unit length. Every ray of a view starts at that camera's center.

Plücker moment. Each ray is stored as its direction d plus its moment m = o × d, where o is the ray's origin. The moment stays the same wherever you pick o along the ray, so (d, m) describes the line itself. Its length is the line's distance from the origin.

Part 2: Mini-LVSM

Q2.1 Overfitting

Final overfit validation PSNR: 30.90 dB (SSIM 0.869).

LVSM overfit visualization

Q2.2 Training and evaluation

Settings: patch size 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm; 2 source views; batch size 4; 100 epochs; AdamW (lr 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps, grad clip 1.0, AMP; MSE loss.

LVSM training curves

Validation visualization at epoch 95 (columns: source views, ground truth, prediction):

LVSM validation visualization epoch 95
Source viewsCorrect PSNRCorrect SSIMShuffled PSNRShuffled SSIMNone PSNRNone SSIM
115.200.55811.810.48712.460.526
216.130.56011.490.47212.440.526
416.860.57511.590.47312.450.536

Source views: with correct rays, PSNR rises steadily with more source views (15.2 → 16.1 → 16.9 dB), even though training used 2. More views cover more of the target, so less has to be hallucinated.

Ray conditioning: without the target rays the model cannot know which view to render. With no rays (zeros) it falls to ≈ 12.4 dB at every S and outputs a blurry, averaged image. Shuffled rays are worse still (≈ 11.5 dB): the model confidently renders the wrong viewpoint. Neither ablation benefits from more source views, which shows the gain at S = 4 comes from geometrically correct conditioning.

Eval S=2 correct rays
S = 2, correct
Eval S=2 shuffled rays
S = 2, shuffled
Eval S=2 no rays
S = 2, none
Eval S=4 correct rays
S = 4, correct

Q2.3 Bonus: loss function, training view count and camera encoding

Question: how do (a) extra loss terms, (b) a random number of training source views, and (c) raw camera parameters in place of Plücker rays change quality? Each run changes one thing from the Q2.2 baseline and keeps every other setting identical (100 epochs, same seed and data).

Validation PSNR and SSIM curves for the LVSM variants
Run (correct rays)S=1 PSNRS=1 SSIMS=2 PSNRS=2 SSIMS=4 PSNRS=4 SSIM
Baseline15.200.55816.130.56016.860.575
+ SSIM loss14.780.57615.440.58015.820.594
+ Perceptual loss13.860.52914.040.52714.150.535
Random S ∈ [1, 4]15.260.55916.270.56017.090.573
Raw camera encoding15.080.55915.910.55816.560.570
SSIM-loss run, S=2
+ SSIM loss, S = 2
Perceptual-loss run, S=2
+ Perceptual loss, S = 2
Random-S run, S=2
Random S, S = 2
Raw camera run, S=2
Raw camera, S = 2

Conclusions.

Limitations: one seed per run, so differences under ≈ 0.2 dB may be noise. Only PSNR/SSIM were measured (no LPIPS), and only on the validation split.

Part 3: Mini-VGGT

Q3.1 Overfitting

After 40 overfit epochs on one scene: total loss −0.150, camera loss 0.0025, depth loss −0.163 (the depth loss is confidence-weighted, so it can go negative).

VGGT overfit loss curve
Overfit scene ground truth
Ground truth
Overfit scene prediction
Prediction (camera + depth)

Q3.2 Training and evaluation

Settings: frozen DINOv2 ViT-B/14 encoder, 224 px; aggregator width 384, 6 heads, 8 layers; 8 views sampled within 32 frames; batch size 16; 1000 epochs; lr 6e-4 with 10 warmup epochs and cosine decay to 0.05×; weight decay 0.01; loss = 5 · camera + 1 · depth. Evaluation: 8 views from the first 32 prepared frames, 224 px, seed 825.

VGGT training loss curve
SplitAbsRel ↓δ1 ↑Rot AUC 5/15/30 ↑Trans AUC 5/15/30 ↑Pose AUC 5/15/30 ↑
Val0.1540.7920.189 / 0.485 / 0.6710.016 / 0.069 / 0.1790.004 / 0.049 / 0.155
Test0.1610.7900.219 / 0.549 / 0.7360.014 / 0.076 / 0.1910.005 / 0.057 / 0.174

Held-out errors: depth holds up well: about 79% of pixels are within 25% of the true depth. The overall room layout matches GT, but surfaces are noisier and edges are blurred. For cameras, rotation is fairly good (mean error ≈ 13°), but translation direction is poor (mean error ≈ 43°, AUC@30 < 0.2), and it drags down the joint pose AUC. The views come from a short stretch of video, so the cameras are close together and even small position errors give large direction errors. Val and test scores are similar, so the model is not overfitting to particular scenes.

Scene 09c1414f1b ground truth
09c1414f1b — GT
Scene 09c1414f1b prediction
09c1414f1b — prediction
Scene 21d970d8de ground truth
21d970d8de — GT
Scene 21d970d8de prediction
21d970d8de — prediction
Scene 280b83fcf3 ground truth
280b83fcf3 — GT
Scene 280b83fcf3 prediction
280b83fcf3 — prediction
Scene 2b1dc6d6a5 ground truth
2b1dc6d6a5 — GT
Scene 2b1dc6d6a5 prediction
2b1dc6d6a5 — prediction

Q3.3 Bonus: multi-task synergy and number of views

(a) Multi-task synergy. Does learning cameras and depth jointly help either task? Two single-task models were trained with the Q3.2 settings, changing only the loss weights: camera-only (camera 5, depth 0) and depth-only (camera 0, depth 1). All three models used the same evaluation protocol. Grey cells are heads that received no training signal.

Model / splitAbsRel ↓δ1 ↑Rot AUC 5/15/30 ↑Trans AUC 5/15/30 ↑Pose AUC 5/15/30 ↑
Joint — val0.1540.7920.189 / 0.485 / 0.6710.016 / 0.069 / 0.1790.004 / 0.049 / 0.155
Camera-only — val0.6450.2690.215 / 0.532 / 0.7080.009 / 0.079 / 0.1950.003 / 0.065 / 0.175
Depth-only — val0.1490.8040.049 / 0.147 / 0.2860 / 0 / 00 / 0 / 0
Joint — test0.1610.7900.219 / 0.549 / 0.7360.014 / 0.076 / 0.1910.005 / 0.057 / 0.174
Camera-only — test0.6530.2690.243 / 0.575 / 0.7570.021 / 0.075 / 0.1990.004 / 0.050 / 0.182
Depth-only — test0.1700.7700.074 / 0.171 / 0.3260 / 0 / 00 / 0 / 0
Camera-only loss curve
Camera-only loss
Depth-only loss curve
Depth-only loss

Conclusions: joint training shows no clear synergy at this scale.

A likely reason is that the frozen DINO features already carry most of the geometry, and the small 8-layer aggregator is capacity-limited, so the tasks compete rather than help each other.

Limitations: single seed; differences are small relative to likely run-to-run variance; only one set of loss weights was tried.

(b) Number of views at evaluation. The Q3.2 model (trained with 8 views) was evaluated with 2, 4, 8, 16 and 32 input views on both splits, also measuring forward time and peak memory.

Validation metrics vs number of views Test metrics vs number of views
ViewsVal AbsRel ↓Val δ1 ↑Val Rot AUC@15Val Pose AUC@30Test AbsRel ↓Test Rot AUC@15Test Pose AUC@30Forward (ms)Peak mem (GB)
20.1120.8670.5400.1350.1190.6260.12433.00.74
40.1370.8370.5080.1450.1470.5350.17132.50.76
80.1540.7920.4850.1550.1610.5500.17532.60.81
160.1920.7340.4480.1320.1710.5550.18134.00.89
320.1980.7210.4520.1320.1770.5680.17640.01.05

Conclusions:

Training with a variable view count would likely be needed to benefit from more views.

Limitations: evaluation-only (no retraining at other view counts). With 2 views there is only one pose pair per sample, so those AUCs are noisy. Metrics at different view counts cover different sets of pixels and pairs.