Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)

All runs use one NVIDIA RTX 4090. Data: ScanNet++ iPhone, 224 × 224 px.

1. Multi-View Data Preprocessing (20 points)

Unprojection (original world frame, meters)

Output of scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000. Scenes 0 and 2 of the five (scene 1 has far outlier points that make its axis box too large). Camera set so that world Z points up, viewed from above and behind the first camera (turned 35° about the up axis so that all three axes show tick labels), with the whole axis box in view.

Unprojected points and cameras in the world frame, scene 0
Unprojected points and cameras in the world frame, scene 2

Canonicalization (first-camera frame, normalized units)

Output of scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000. Same view direction as above (the world up direction is mapped into the first-camera frame). The axes show that the origin is at the first camera and the scale is in normalized units. The text below the depth image shows the first camera center and the valid-point centroid distance.

Canonicalized points and cameras, scene 0
Canonicalized points and cameras, scene 2

check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 8.228723163483664e-05,
      "max_ray_reprojection_error_px": 8.3662846009247e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 9.281559323426336e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.5367431640625e-05,
      "max_ray_reprojection_error_px": 8.217109279939905e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011776237079175189,
      "max_ray_reprojection_error_px": 0.00011528912727953866,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. Canonicalization uses the camera frame of the first image as the world frame: the first camera has the identity pose, and every other camera is a transformation relative to this reference frame. It also divides all translations, depths and points by one scene scale, so that the valid-point centroid is at distance 1 from the origin.

World-space ray directions. The world-space ray direction of a pixel is the unit direction, in the canonical world frame, of the ray from the camera center through that pixel.

Plücker moment. The Plücker moment is the cross product of a point on the ray (here the ray origin, the camera center) and the ray direction: m = o × d. Any other point on the same ray gives the same moment, so (d, m) describes the ray itself, not the point on the ray that was used as its origin.

2.1 Overfitting (10 points)

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit: 2000 steps on one fixed batch (4 samples, S = 2 source views, 6 target views each).

Final overfit PSNR: 30.92 dB (SSIM 0.870).

Columns: source views, ground truth, prediction.

Overfit visualization: source views, ground truth and prediction

2.2 Training and Evaluation (30 points)

Training settings

Training curves

Mini-LVSM training and validation loss, PSNR and SSIM

Validation PSNR / SSIM for each ablation setting

Sources sit at views {0}, {0, 7} and {0, 2, 5, 7} for S = 1, 2, 4. Solid lines and "shared" columns: only views {1, 3, 4, 6}, which are targets for every S (112 images for each setting), so all S are scored on the same images. Dashed lines and "all" columns: standard evaluation with all 8 − S target views (196, 168 and 112 images).

PSNR and SSIM against the number of source views, one color for each conditioning; solid lines for shared target views, dashed lines for all target views
Source views STarget-ray conditioningPSNR shared (dB)PSNR all (dB)SSIM sharedSSIM allImages shared / all
1correct15.2315.200.5660.559112 / 196
1shuffled11.7911.800.4960.487112 / 196
1none12.4112.460.5330.528112 / 196
2correct16.2416.140.5720.560112 / 168
2shuffled11.5511.490.4810.472112 / 168
2none12.4312.430.5360.526112 / 168
4correct16.8716.870.5750.575112 / 112
4shuffled11.5911.590.4730.473112 / 112
4none12.4412.440.5370.537112 / 112

Training visualizations (validation, during training)

The same validation sample at each epoch. Columns: the 2 source views, ground truth, then the prediction at epoch 0, 50 and 95. Rows: target views.

Validation sample: source views, ground truth and predictions at epoch 0, 50 and 95

Evaluation visualizations

One figure for each S. In each figure, the source views and ground truth are the same for all conditionings. Columns: source views, ground truth, then the prediction with correct, shuffled and none conditioning. Rows: target views.

S = 1

Evaluation, S=1: source views, ground truth, predictions for correct, shuffled and none conditioning

S = 2

Evaluation, S=2: source views, ground truth, predictions for correct, shuffled and none conditioning

S = 4

Evaluation, S=4: source views, ground truth, predictions for correct, shuffled and none conditioning

Discussion

Number of source views. With correct ray conditioning, more source views give predictions closer to the ground truth: PSNR and SSIM increase from S = 1 to S = 4. With shuffled or no target rays, more source views do not help.

Ray conditioning. Correct target rays give the highest PSNR and SSIM, no rays (none) give lower values, and shuffled (wrong) rays give the lowest. This is expected: correct rays give the network the correct condition, none means a prediction without a pose condition, and shuffled means a wrong condition.

Which views are scored. The source views are at fixed positions in the view list: {0}, {0, 7} and {0, 2, 5, 7} for S = 1, 2, 4. The loader puts the sampled frames in random order, so these positions are random frames, but the set of target views changes with S: S = 1 also scores views 2, 5 and 7, and S = 2 also scores views 2 and 5. Each target view has its own difficulty, so the mean over all 8 − S targets mixes the effect of S with the effect of which views are scored. For this reason, the solid lines and the "shared" columns compare S = 1, 2, 4 on only the views that all of them score, {1, 3, 4, 6}.

2.3 Bonus: Analysis and Exploration (10 points)

Question

Training view count: does a random number of source views per training step change evaluation with 1, 2 and 4 source views?

Experimental settings

Results

Same target views {1, 3, 4, 6} for every S.

SplitSPSNR fixed S=2PSNR random SSSIM fixed S=2SSIM random S
val115.3215.310.56800.5706
val216.3916.380.57360.5754
val417.0317.140.57700.5776
test115.4615.450.53450.5372
test216.3616.350.53950.5398
test417.2017.330.54440.5436

Standard evaluation (models.lvsm.eval), all 8 − S target views.

SplitSPSNR fixed S=2PSNR random SSSIM fixed S=2SSIM random S
val115.2815.290.5610.563
val216.2716.260.5620.563
val417.0317.140.5770.578
test115.6115.570.5390.541
test216.3616.330.5450.544
test417.2017.330.5440.544

PSNR and SSIM against the number of source views, validation and test; solid lines for target views 1, 3, 4, 6, dashed lines for all target views
Validation PSNR at S=2 during training for both runs

Paired comparison, random S minus fixed S = 2: per-sample mean difference over the 4 common target views {1, 3, 4, 6} of each of the 28 samples; 95% interval from 5000 bootstrap resamples of the samples. Wins = samples where random S is higher.

SplitSΔPSNR (dB) [95% CI]Wins / 28ΔSSIM [95% CI]Wins / 28
val1-0.00 [-0.05, +0.05]13+0.0026 [+0.0009, +0.0042]20
val2-0.02 [-0.06, +0.03]12+0.0019 [+0.0001, +0.0037]16
val4+0.11 [+0.04, +0.20]17+0.0006 [-0.0015, +0.0027]12
test1-0.01 [-0.06, +0.04]17+0.0027 [+0.0009, +0.0044]21
test2-0.01 [-0.06, +0.04]14+0.0003 [-0.0014, +0.0020]16
test4+0.13 [+0.06, +0.20]22-0.0008 [-0.0028, +0.0013]12

Validation predictions, correct conditioning. One figure for each S. In each figure, the source views and ground truth are the same for both models. Columns: source views, ground truth, then the prediction of the fixed S = 2 model and of the random S model. Rows: target views.

S = 1

S=1: source views, ground truth, predictions of the fixed S=2 and random S models

S = 2

S=2: source views, ground truth, predictions of the fixed S=2 and random S models

S = 4

S=4: source views, ground truth, predictions of the fixed S=2 and random S models

Conclusions

The model trained with a random number of source views (S drawn from {1, 2, 3, 4}) performs similarly to the model trained with S = 2. Random S is slightly better in some settings (see Limitations).

Possible explanations

With random S, the mean number of source views during training is 2.5, a little higher than 2. The Q2.2 ablation shows that more source views at test time improve the result. This is intuitive: more source views cover a larger part of the space, so a larger part of the target view overlaps a source view and is grounded by it. A higher mean S during training can help in a similar way.

Limitations

The higher mean S during training does not improve PSNR and SSIM at every test-time S. On the shared target views, random S gives a higher PSNR only at S = 4, and a higher SSIM only at S = 1 and at S = 2 on validation. The other differences are inside the 95% intervals. The improvements are too small to see in the images.

3.1 Overfitting (10 points)

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit: one fixed training clip, 8 views, 40 epochs × 25 steps, AdamW, learning rate 1e-3 with cosine decay to 5%, no warmup. Loss = 5 × camera + 1 × depth.

Overfit loss curves: total, camera and depth loss

Ground truth (left) and prediction (right), same clip, canonical first-camera coordinates. Red frustum: view 0. View from the ceiling, orthographic: the viewer camera looks straight down the gravity axis, estimated as the normal of the largest horizontal plane in the GT points (floor or table tops). The first camera looks up the image. Legend and title hidden, points drawn larger.

Overfit clip, ground-truth geometry, top view Overfit clip, predicted geometry, top view

3.2 Training and Evaluation (30 points)

Training settings

Training loss curves: total, camera and depth loss

Validation and test metrics

Evaluation of best.pt: 8 views from the first 32 prepared frames, 224 px, seed 825. 28 samples per split.

MetricValidationTest
Depth AbsRel (lower is better)0.1330.156
Depth δ10.8480.830
Rotation AUC@50.2480.260
Rotation AUC@150.5810.609
Rotation AUC@300.7420.778
Translation AUC@50.0370.015
Translation AUC@150.1040.090
Translation AUC@300.2090.221
Joint pose AUC@50.0140.004
Joint pose AUC@150.0820.071
Joint pose AUC@300.1850.204

GT and predicted geometry, four validation scenes

Left: ground truth. Right: prediction (predicted cameras and depth, unprojected into the first-camera frame). Same view from the ceiling as in Q3.1.

Scene 09c1414f1b

Scene 09c1414f1b, ground truth, top view Scene 09c1414f1b, prediction, top view

Scene 21d970d8de

Scene 21d970d8de, ground truth, top view Scene 21d970d8de, prediction, top view

Scene 280b83fcf3

Scene 280b83fcf3, ground truth, top view Scene 280b83fcf3, prediction, top view

Scene 2b1dc6d6a5

Scene 2b1dc6d6a5, ground truth, top view Scene 2b1dc6d6a5, prediction, top view

Discussion

Depth of view 0 for the four validation scenes: GT, prediction and per-pixel AbsRel. Magenta: pixels without valid GT depth (not scored).

View 0 RGB, GT depth, predicted depth and AbsRel for four validation scenes

Predicted (orange) and GT (gray) cameras of the same four samples in the first-camera frame, without alignment (view 0 is the identity in both). Top: top view (X–Z). Bottom: side view (Z–Y). Dotted lines join the GT and predicted centers of the same view. The text gives the rotation error of views 1–7.

Predicted and GT camera frustums for four validation scenes, top and side views

Depth. The largest depth errors are near large depth discontinuities. Near an edge, the model tends to average the depths of the two surfaces, so in the predicted depth maps the edges are blurred and show a gradient instead of a step. In the point clouds, this puts points between the foreground and the background surface.

Small camera baseline. The cameras of one sample are close together: the 8 views come from a range of 32 consecutive frames. A new view therefore adds little parallax, and the depth does not get much correction from the other views.

Camera pose. Most of the camera pose error comes from translation (translation AUC is much lower than rotation AUC). This can be related to the limited depth understanding of the model.

3.3 Bonus: Analysis and Exploration (10 points)

Exploration A: Point-map prediction

Question

Is a point map from a dedicated point head better than points from the predicted depth and cameras? Does the extra point head change the camera and depth results?

Experimental settings

Results

SplitPointsL2 (normalized)L2 (m)Inlier@0.05Inlier@0.1Inlier@0.2
valBaseline, unprojected0.2390.4830.1330.3330.631
valPoint-head model, unprojected0.2260.4600.1340.3540.640
valPoint-head model, direct0.2270.4640.1150.3370.638
testBaseline, unprojected0.2150.4050.1160.3220.652
testPoint-head model, unprojected0.1770.3420.1490.4140.746
testPoint-head model, direct0.1880.3600.1070.3760.726

Point inlier share for the three point sources, validation and test

Camera and depth metrics of best.pt (point-head model: epoch 998).

Depth AbsRel, depth delta1 and rotation, translation and joint pose AUC of the baseline and the point-head model, validation and test
MetricBaseline valPoint head valBaseline testPoint head test
Depth AbsRel (lower is better)0.1330.1340.1560.127
Depth δ10.8480.8520.8300.859
Rotation AUC@50.2480.2580.2600.262
Rotation AUC@150.5810.5890.6090.597
Rotation AUC@300.7420.7490.7780.775
Translation AUC@50.0370.0230.0150.018
Translation AUC@150.1040.1000.0900.116
Translation AUC@300.2090.2250.2210.247
Joint pose AUC@50.0140.0110.0040.005
Joint pose AUC@150.0820.0870.0710.093
Joint pose AUC@300.1850.2070.2040.230

Point-head model loss curves.

Point-head model loss curves

Four validation scenes, same view from the ceiling as in Q3.2. Left to right: ground truth, point-head model with points from predicted depth and cameras, point-head model with points from the point head.

Scene 09c1414f1b

Scene 09c1414f1b, ground truth Scene 09c1414f1b, points from depth and cameras Scene 09c1414f1b, points from the point head

Scene 21d970d8de

Scene 21d970d8de, ground truth Scene 21d970d8de, points from depth and cameras Scene 21d970d8de, points from the point head

Scene 280b83fcf3

Scene 280b83fcf3, ground truth Scene 280b83fcf3, points from depth and cameras Scene 280b83fcf3, points from the point head

Scene 2b1dc6d6a5

Scene 2b1dc6d6a5, ground truth Scene 2b1dc6d6a5, points from depth and cameras Scene 2b1dc6d6a5, points from the point head

Conclusions

The point head improves the depth result and, on the test split, the camera result. On validation the changes are small and mixed.

In the point clouds of the four validation scenes, the point-head model puts the points that are far from the camera closer to the GT. The point error binned by GT depth (scripts/vggt_point_error_by_depth.py) agrees on the validation split: the error is mostly lower beyond 2 m and almost the same nearer than 2 m. On the test split the gain is at the near points (nearer than 2 m), and points beyond 3 m have the same or a slightly larger error.

Possible explanations

The point head adds a second supervision signal for the same geometry. Its targets are world points in the first-camera frame, which depend on the depth and the camera pose together, so its gradients bring information about both into the shared backbone. This extra multi-task supervision can make the shared features better for the depth and camera heads.

Limitations

The point head does not fix the depth averaging at depth discontinuities: the point clouds of the point-head model still put many points between the foreground and the background surface. Camera translation also stays a problem: the translation AUC of the point-head model is still much lower than its rotation AUC.

Exploration B: Number of views

Question

How does the number of input views change accuracy, runtime and memory? Does training with a random number of views change the results?

Experimental settings

Results

SplitVModelRot AUC@30 [95% CI]Trans AUC@30 [95% CI]AbsRel [95% CI]δ1 [95% CI]Joint AUC@30
val2Baseline0.768 [0.654, 0.861]0.238 [0.126, 0.351]0.128 [0.095, 0.178]0.860 [0.773, 0.920]0.236
val2Random views0.723 [0.621, 0.814]0.236 [0.125, 0.358]0.111 [0.089, 0.137]0.864 [0.785, 0.924]0.201
val4Baseline0.725 [0.641, 0.796]0.220 [0.155, 0.288]0.133 [0.095, 0.184]0.831 [0.725, 0.914]0.191
val4Random views0.699 [0.622, 0.769]0.201 [0.141, 0.269]0.130 [0.101, 0.171]0.861 [0.784, 0.917]0.183
val8Baseline0.742 [0.663, 0.814]0.209 [0.154, 0.272]0.133 [0.098, 0.186]0.848 [0.765, 0.911]0.185
val8Random views0.712 [0.637, 0.774]0.183 [0.132, 0.246]0.146 [0.109, 0.198]0.824 [0.733, 0.897]0.169
test2Baseline0.799 [0.708, 0.871]0.181 [0.080, 0.290]0.106 [0.090, 0.122]0.881 [0.835, 0.919]0.168
test2Random views0.729 [0.627, 0.824]0.230 [0.119, 0.345]0.111 [0.090, 0.134]0.884 [0.829, 0.931]0.198
test4Baseline0.786 [0.727, 0.834]0.222 [0.153, 0.290]0.139 [0.103, 0.193]0.831 [0.756, 0.891]0.202
test4Random views0.776 [0.713, 0.826]0.191 [0.127, 0.262]0.133 [0.100, 0.182]0.845 [0.756, 0.909]0.168
test8Baseline0.778 [0.719, 0.826]0.221 [0.165, 0.276]0.156 [0.112, 0.230]0.830 [0.744, 0.896]0.204
test8Random views0.747 [0.688, 0.799]0.183 [0.131, 0.239]0.147 [0.106, 0.204]0.818 [0.732, 0.887]0.169

Accuracy against the number of evaluation views with 95% bootstrap intervals

Same evaluated outputs for every V. In the table above, depth is scored over views 0 to V − 1 and pose over 1, 6 or 28 pairs per sample. The view selection is nested, so views 0 and 1 are the same frames for every V. The table below scores only the depth of views 0 and 1 and the pose of pair (0, 1) (one pair per sample). The scene scale still uses all V input views. 95% intervals from 2000 bootstrap resamples of the 28 samples.

SplitVModelRot AUC@30, pair (0, 1)Trans AUC@30, pair (0, 1)AbsRel, views 0–1δ1, views 0–1
val2Baseline0.768 [0.659, 0.858]0.238 [0.132, 0.355]0.128 [0.095, 0.176]0.860 [0.776, 0.921]
val2Random views0.723 [0.614, 0.818]0.236 [0.125, 0.348]0.111 [0.088, 0.136]0.864 [0.788, 0.929]
val4Baseline0.773 [0.660, 0.862]0.320 [0.206, 0.437]0.131 [0.093, 0.183]0.840 [0.729, 0.919]
val4Random views0.740 [0.638, 0.830]0.255 [0.144, 0.364]0.126 [0.097, 0.169]0.871 [0.790, 0.927]
val8Baseline0.754 [0.638, 0.858]0.333 [0.213, 0.448]0.124 [0.088, 0.176]0.874 [0.789, 0.934]
val8Random views0.721 [0.618, 0.812]0.235 [0.130, 0.356]0.136 [0.099, 0.193]0.842 [0.742, 0.916]
test2Baseline0.799 [0.715, 0.873]0.181 [0.081, 0.289]0.106 [0.090, 0.122]0.881 [0.836, 0.921]
test2Random views0.729 [0.624, 0.824]0.230 [0.124, 0.342]0.111 [0.090, 0.135]0.884 [0.827, 0.933]
test4Baseline0.805 [0.717, 0.873]0.236 [0.131, 0.352]0.128 [0.098, 0.170]0.853 [0.779, 0.909]
test4Random views0.788 [0.698, 0.869]0.227 [0.111, 0.348]0.126 [0.096, 0.160]0.846 [0.767, 0.915]
test8Baseline0.800 [0.715, 0.871]0.270 [0.156, 0.392]0.145 [0.107, 0.203]0.830 [0.734, 0.900]
test8Random views0.763 [0.671, 0.840]0.221 [0.120, 0.344]0.139 [0.107, 0.184]0.835 [0.744, 0.903]
Views VForward time (ms)Time per view (ms)Peak memory (MiB)
210.65.3709
410.52.6727
813.41.7775
1620.81.3856
3240.31.31016

Forward time and peak GPU memory against the number of views

Random-view model loss curves.

Random-view model loss curves

Four validation scenes at 8 views, same view from the ceiling as in Q3.2. Left to right: ground truth, baseline, random-view model.

Scene 09c1414f1b

Scene 09c1414f1b, ground truth Scene 09c1414f1b, baseline Scene 09c1414f1b, random-view model

Scene 21d970d8de

Scene 21d970d8de, ground truth Scene 21d970d8de, baseline Scene 21d970d8de, random-view model

Scene 280b83fcf3

Scene 280b83fcf3, ground truth Scene 280b83fcf3, baseline Scene 280b83fcf3, random-view model

Scene 2b1dc6d6a5

Scene 2b1dc6d6a5, ground truth Scene 2b1dc6d6a5, baseline Scene 2b1dc6d6a5, random-view model

Conclusions

The random-view model is slightly worse than the baseline, mostly in camera pose: its rotation AUC@30 is lower at every V on both splits, and its translation AUC@30 is lower at V = 4 and 8. A possible reason is that the mean number of views during training decreases: k is drawn uniformly from 2 to 8, so the model sees 5 views per sample on average instead of 8. The depth results are mixed, and all differences are inside the 95% intervals.