16-825 Assignment 4: Multi-View Transformers

Andrew ID: mehark

All numbers are on the given ScanNet++ subset at 224×224 resolution.

1. Multi-view data preprocessing

Unprojection (Q1.1, Q1.2)

Points and cameras in the original world frame (in meters) for two training scenes.

Interactive version (two scenes, 1500 points per view, drag to rotate and scroll to zoom):

Canonicalization (Q1.3)

The same frames after canonicalization. The text under the depth image gives the first camera center, which is at the origin, and the valid-point centroid distance, which is 1.000.

Interactive version:

Final check (Q1.7)

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

Output of the check (four training samples, batch size 2):

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011280165199423209,
      "max_ray_reprojection_error_px": 0.00010225349979009479,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Both reprojection errors are about 100 times below the 0.01 px limit and the centroid distance is 1 in every sample.

Explanations

Canonicalization. Each ScanNet++ scene comes in its own arbitrary world frame and its own size. All poses are multiplied by the inverse of the first camera's pose, so camera 0 becomes the identity. Then, the translations, depth and points are divided by the distance from origin to the centroid of the valid points. Consequently every sample looks the same to the a model: camera 0 at the origin looking down +Z axis, with the average point about one unit away. The centroid is not subtracted, so camera 0 stays exactly at the origin.

World-space ray directions. For a pixel (u, v), K⁻¹ [u, v, 1]ᵀ is its ray direction in the camera frame. Rotating it by the rotation part of the camera-to-world pose, then normalizing gives the unit direction in the canonical frame. The origin of every ray of a view is that view's camera center.

Plücker moment. The moment is m = o x d, the cross product of the ray origin and its direction. Together (d, m) describe the line itself rather than a point on it: sliding the origin along the ray, o + λd, gives the same m. The moment is perpendicular to d and the lenght is the distance of the line from the world origin.

2. Novel-view synthesis with Mini-LVSM

2.1 Overfitting

After 2000 steps on one fixed batch the model reaches 31.01 dB PSNR (SSIM 0.872, MSE 0.00085) on that batch. Columns: the two source views, ground truth, prediction.

2.2 Training and evaluation

Training settings (the provided configs/lvsm.json, unchanged): patch size 8, width 384, 8 layers, 14.6M parameters; 2 source views and 6 target views per sample, sampled from a window of 32 frames; batch size 4; AdamW with learning rate 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warm-up steps then cosine decay; gradient clipping at 1.0; bfloat16 autocast; 100 epochs (5600 steps) (21.5 minutes on 1x GPU). I also ran exactly the same config for 500 epochs (115 minutes) to see how much of the result is the short schdule.

Validation PSNR (dB) / SSIM for every ablation setting, 100 epochs:

Source views correct rays shuffled rays no rays
1 15.20 / 0.558 11.81 / 0.488 12.48 / 0.528
2 16.14 / 0.560 11.50 / 0.472 12.46 / 0.526
4 16.88 / 0.575 11.61 / 0.474 12.46 / 0.537

The same model trained for 500 epochs:

Source views correct rays shuffled rays no rays
1 15.97 / 0.592 11.58 / 0.479 12.42 / 0.534
2 17.34 / 0.605 11.18 / 0.457 12.40 / 0.531
4 18.57 / 0.628 11.24 / 0.453 12.38 / 0.541

Training-time visualization (validation batch at epoch 95 of the 100-epoch run, columns: source views, ground truth, prediction):

Evaluation visualizations, 100-epoch model with 2 source views (columns: the two source views, ground truth, prediction):

Correct target rays

Shuffled target rays

No target rays

500-epoch model with 4 source views and correct rays:

Improved training settings. After the bonus experiment below showed that training with 4 source views (+ 4 target views; as opposed to the default 2 source views + 6 target views) helps, I trained the default model with 4 source views for 420 and for 600 epochs (134 and 172 minutes on 1x GPU). Same eval on the same validation views:

Model Test: 1 source view Test: 2 source views Test: 4 source views
default, 2 views, 100 epochs (handout config) 15.20 / 0.558 16.14 / 0.560 16.88 / 0.575
default, 2 views, 500 epochs 15.97 / 0.592 17.34 / 0.605 18.57 / 0.628
4 views, 420 epochs 16.02 / 0.596 17.55 / 0.616 19.10 / 0.648
4 views, 600 epochs 16.12 / 0.601 17.74 / 0.623 19.43 / 0.658

The best model (4 views, 600 epochs) with 4 source views and correct rays:

That is +2.5 dB over the defualt config with 4 source views and +1.6 dB with 2, and the ray ablation still holds for it (11.1 dB with shuffled rays, 12.4 dB with none). The images are a little sharper, the chairs and the lamp are now recognizable shapes instead of blobs, but they are still soft, and validation PSNR was still rising slowly at epoch 600, so this model can seemingly converge further.

Number of source views. PSNR goes up with every extra source view (15.2 → 16.1 → 16.9 dB for the 100-epoch model, 16.0 → 17.3 → 18.6 dB for the 500-epoch one), even though the default model was only ever trained with two. More sources mean more of the target view has actually been observed, and since every token attends to all source tokens the model can, seemingly without complaint, use them without ever having been retrained for it. SSIM moves much less than PSNR, because the outputs are blurry either way.

Ray conditioning. With the rays of a different target view (shuffled) PSNR drops to about 11.5 dB, and with all-zero rays to about 12.4 dB, 3 to 7 dB below the correct rays. Clearly the model really does use the target camera. Shuffled is worse than none: with no rays the model falls back to an average of the sources, with wrong rays it confidently draws a different view. More source views do not help in either case.

2.3 Bonus: how many source views to train wiht

Question. The baseline is trained with exactly 2 source views. Does the number of source views used in training change how well the model does when it is tested with 1, 2 and 4?

Setup. Four trainings that differ only in the source views per step: always 1, always 2 (the baseline), always 4, and a random choice of 1, 2 or 4 at every step (train_source_view_choices in configs/lvsm_random_views.json). Everything else is the default config, 100 epochs each. The remaining views of the 8 are the targets, so training with more sources also means fewer target images per step. To know how large the seed noise is, the 2-view and 4-view models were trained a second time with another seed. All models are evaluated on the same validation views.

Validation PSNR (dB) / SSIM with correct rays:

Trained with test: 1 source view test: 2 source views test: 4 source views
1 source view 15.12 / 0.559 14.90 / 0.552 14.64 / 0.561
2 source views (baseline) 15.20 / 0.558 16.14 / 0.560 16.88 / 0.575
random 1, 2 or 4 per step 15.27 / 0.561 16.25 / 0.561 17.10 / 0.575
4 source views 15.31 / 0.556 16.41 / 0.561 17.46 / 0.578

Results. A second seed of the 2-view and 4-view models moved the numbers by at most 0.07 dB (hollow markers in the plot), so differences of a few tenths of a dB are real. Training with 4 source views is the best in every column: +0.25 dB over the baseline when tested with 2 views and +0.5 dB with 4, and no worse with 1 (that gap is inside the noise). The random-count model sits between the 2-view and the 4-view model. The model trained with a single source view is clearly the odd one out. It gets worse when it is given more source views at test time (15.1 → 14.9 → 14.6 dB), which is a strange thing to watch happen to a model that was given more information.

Explanation. A model that only ever saw one source never had to decide which of several sources to trust for a given target patch, so extra sources at test time are just a longer, unfamiliar sequence. Two sources are enough to learn that, and the skill transfers to four. Four sources in training help even at test time with fewer, supposedly because each target is easier to explain, so the gradient is less dominated by regions no source has seen and the model learns the geometry of matching rays faster.

Limitations. The comparison is at equal epochs, not equal compute (the 4-view model is about 25% slower per epoch), all runs are short, and there is only a small separation in the numbers. As for the method, each target is rendered from its own (S + 1)·N-token sequence with full attention, so cost per target grows quadratically with the number of source views.

3. Multi-view geometry with Mini-VGGT

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

One fixed traiinng clip, 40 epochs of 25 steps, learning rate 1e-3. The total loss goes from 5.48 to −0.16 (it can be negative because the confidence term subtracts 0.2log of the confidence).

Ground truth:

Prediction:

Interactive version (the geometry drop-down switches between ground truth and the prediction):

3.2 Training and evaluation

Training settings (default setting): frozen DINOv2 vitb/14 encoder, 8 frame/global attention pairs of width 384 with 6 heads; 8 views per clip from a 32-frame window; batch size 16; learning rate 6e-4 with 10 warm-up epochs and cosine decay to 5%, weight decay 0.01; loss weights 5 for the camera and 1 for the depth; 1000 epochs (14,000 steps). It ran on 4x GPUs with torchrun (global batch size still 16) and took 85 mintues. The best checkpoint by validation loss is from epoch 852.

Metrics of best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825:

Metric validation test
Depth AbsRel 0.141 0.153
Depth δ1 0.841 0.814
Rotation AUC @5° / 15° / 30° 0.187 / 0.499 / 0.689 0.206 / 0.525 / 0.727
Translation AUC @5° / 15° / 30° 0.013 / 0.071 / 0.173 0.012 / 0.054 / 0.161
Joint pose AUC @5° / 15° / 30° 0.002 / 0.052 / 0.150 0.002 / 0.034 / 0.149

Ground truth (first image) and predicted camera + depth (second image) for four validation scenes.

Interactive version for the first two validation scenes (ground truth / prediction in the geometry drop-down):

Errors on held-out scenes. Depth generalizes best: 84% of the pixels are within a factor 1.25 of the ground truth on validation and 81% on test, but the point clouds are smooth and blobby, with thin structures and sharp depth edges clearly lost. Rotations are roughly right (AUC@30 about 0.7, mean error 12° on validation and 9° on test) but not precise (AUC@5 about 0.2). The translation direction is the weak point: mean error about 44°, AUC@30 only 0.17. The cameras in these clips move very little compared with the distance to the scene (baselines around 0.1 to 0.3 normalized units). Hence the L1 loss on the translation is small whatever direction is predicted, and the model seemingly settles for learning its size. The joint pose AUC is limited by the translation.

3.3 Bonus: number of input views

Question. The model is trained and evaluated with 8 views. What happens to accuracy, runtime and memory when it gets a different number of views at test time, and does a model trained with fewer views behave differently?

Setup. Two models with the default config that differ only in the views per training clip: 8 (the baseline of 3.2) and 4 (--num-views 4, 43 minutes of training instead of 85). Both are evaluated with 2 to 32 views taken from the first 32 prepared frames of each validation scene. Forward time is the average over 10 batches of 4 clips after two warm-up batches, and memory is the peak allocted on the GPU. In each cell, first number is the model trained with 8 views and the second the model trained with 4.

Validation split:

Test views AbsRel (8 / 4) δ1 (8 / 4) Rotation AUC@30 (8 / 4) Translation AUC@30 (8 / 4) Joint AUC@30 (8 / 4) ms per 4 clips Peak memory (GB)
2 0.126 / 0.111 0.852 / 0.867 0.669 / 0.689 0.163 / 0.221 0.136 / 0.179 33 0.75
3 0.128 / 0.128 0.857 / 0.874 0.637 / 0.672 0.187 / 0.170 0.154 / 0.141 31 0.77
4 0.129 / 0.125 0.856 / 0.879 0.663 / 0.677 0.170 / 0.140 0.140 / 0.116 32 0.82
6 0.132 / 0.134 0.859 / 0.861 0.682 / 0.681 0.160 / 0.145 0.135 / 0.124 33 0.88
8 0.141 / 0.146 0.841 / 0.832 0.689 / 0.685 0.172 / 0.163 0.149 / 0.140 36 0.95
12 0.155 / 0.167 0.802 / 0.794 0.682 / 0.669 0.152 / 0.148 0.131 / 0.127 50 1.09
16 0.168 / 0.182 0.794 / 0.786 0.658 / 0.656 0.154 / 0.146 0.132 / 0.127 67 1.23
24 0.167 / 0.185 0.783 / 0.777 0.664 / 0.660 0.160 / 0.146 0.136 / 0.128 112 1.50
32 0.172 / 0.191 0.774 / 0.764 0.651 / 0.653 0.153 / 0.146 0.128 / 0.126 165 1.78

Results.

Explanation. Global attention runs with any number of views, but the network was only trained on sequences of one length: with 16 or 32 views every token attends over two to four times as many tokens as in training. Consequently depth, which is decoded per patch from those features, suffers first. The data also changes with the view count: the scene scale is recomputed from the selected views, and two views give a single camera pair per scene (28 pairs in total), which is why the 2-view pose numbers jump around.

Limitations. Global attention over all V·261 tokens, so compute grows quadratically with the number of views. Also the model does not generalzie to greater view counts.