16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)
All runs use one NVIDIA RTX 4090. Data: ScanNet++ iPhone, 224 × 224 px.
Output of scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000. Scenes 0 and 2 of the five (scene 1 has far outlier points that make its axis box too large). Camera set so that world Z points up, viewed from above and behind the first camera (turned 35° about the up axis so that all three axes show tick labels), with the whole axis box in view.


Output of scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000. Same view direction as above (the world up direction is mapped into the first-camera frame). The axes show that the origin is at the first camera and the scale is in normalized units. The text below the depth image shows the first camera center and the valid-point centroid distance.


check_data.py outputpython scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 8.228723163483664e-05,
"max_ray_reprojection_error_px": 8.3662846009247e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 9.281559323426336e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.5367431640625e-05,
"max_ray_reprojection_error_px": 8.217109279939905e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011776237079175189,
"max_ray_reprojection_error_px": 0.00011528912727953866,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization. Canonicalization uses the camera frame of the first image as the world frame: the first camera has the identity pose, and every other camera is a transformation relative to this reference frame. It also divides all translations, depths and points by one scene scale, so that the valid-point centroid is at distance 1 from the origin.
World-space ray directions. The world-space ray direction of a pixel is the unit direction, in the canonical world frame, of the ray from the camera center through that pixel.
Plücker moment. The Plücker moment is the cross product of a point on the ray (here the ray origin, the camera center) and the ray direction: m = o × d. Any other point on the same ray gives the same moment, so (d, m) describes the ray itself, not the point on the ray that was used as its origin.
python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit: 2000 steps on one fixed batch (4 samples, S = 2 source views, 6 target views each).
Final overfit PSNR: 30.92 dB (SSIM 0.870).
Columns: source views, ground truth, prediction.
configs/lvsm.json.best.pt = best validation PSNR at S = 2 (epoch 99).
Sources sit at views {0}, {0, 7} and {0, 2, 5, 7} for S = 1, 2, 4. Solid lines and "shared" columns: only views {1, 3, 4, 6}, which are targets for every S (112 images for each setting), so all S are scored on the same images. Dashed lines and "all" columns: standard evaluation with all 8 − S target views (196, 168 and 112 images).
| Source views S | Target-ray conditioning | PSNR shared (dB) | PSNR all (dB) | SSIM shared | SSIM all | Images shared / all |
|---|---|---|---|---|---|---|
| 1 | correct | 15.23 | 15.20 | 0.566 | 0.559 | 112 / 196 |
| 1 | shuffled | 11.79 | 11.80 | 0.496 | 0.487 | 112 / 196 |
| 1 | none | 12.41 | 12.46 | 0.533 | 0.528 | 112 / 196 |
| 2 | correct | 16.24 | 16.14 | 0.572 | 0.560 | 112 / 168 |
| 2 | shuffled | 11.55 | 11.49 | 0.481 | 0.472 | 112 / 168 |
| 2 | none | 12.43 | 12.43 | 0.536 | 0.526 | 112 / 168 |
| 4 | correct | 16.87 | 16.87 | 0.575 | 0.575 | 112 / 112 |
| 4 | shuffled | 11.59 | 11.59 | 0.473 | 0.473 | 112 / 112 |
| 4 | none | 12.44 | 12.44 | 0.537 | 0.537 | 112 / 112 |
The same validation sample at each epoch. Columns: the 2 source views, ground truth, then the prediction at epoch 0, 50 and 95. Rows: target views.
One figure for each S. In each figure, the source views and ground truth are the same for all conditionings. Columns: source views, ground truth, then the prediction with correct, shuffled and none conditioning. Rows: target views.
S = 1
S = 2
S = 4
Number of source views. With correct ray conditioning, more source views give predictions closer to the ground truth: PSNR and SSIM increase from S = 1 to S = 4. With shuffled or no target rays, more source views do not help.
Ray conditioning. Correct target rays give the highest PSNR and SSIM, no rays (none) give lower values, and shuffled (wrong) rays give the lowest. This is expected: correct rays give the network the correct condition, none means a prediction without a pose condition, and shuffled means a wrong condition.
Which views are scored. The source views are at fixed positions in the view list: {0}, {0, 7} and {0, 2, 5, 7} for S = 1, 2, 4. The loader puts the sampled frames in random order, so these positions are random frames, but the set of target views changes with S: S = 1 also scores views 2, 5 and 7, and S = 2 also scores views 2 and 5. Each target view has its own difficulty, so the mean over all 8 − S targets mixes the effect of S with the effect of which views are scored. For this reason, the solid lines and the "shared" columns compare S = 1, 2, 4 on only the views that all of them score, {1, 3, 4, 6}.
Training view count: does a random number of source views per training step change evaluation with 1, 2 and 4 source views?
best.pt of each run is evaluated at S = 1, 2, 4 with correct conditioning, on validation and test.Same target views {1, 3, 4, 6} for every S.
| Split | S | PSNR fixed S=2 | PSNR random S | SSIM fixed S=2 | SSIM random S |
|---|---|---|---|---|---|
| val | 1 | 15.32 | 15.31 | 0.5680 | 0.5706 |
| val | 2 | 16.39 | 16.38 | 0.5736 | 0.5754 |
| val | 4 | 17.03 | 17.14 | 0.5770 | 0.5776 |
| test | 1 | 15.46 | 15.45 | 0.5345 | 0.5372 |
| test | 2 | 16.36 | 16.35 | 0.5395 | 0.5398 |
| test | 4 | 17.20 | 17.33 | 0.5444 | 0.5436 |
Standard evaluation (models.lvsm.eval), all 8 − S target views.
| Split | S | PSNR fixed S=2 | PSNR random S | SSIM fixed S=2 | SSIM random S |
|---|---|---|---|---|---|
| val | 1 | 15.28 | 15.29 | 0.561 | 0.563 |
| val | 2 | 16.27 | 16.26 | 0.562 | 0.563 |
| val | 4 | 17.03 | 17.14 | 0.577 | 0.578 |
| test | 1 | 15.61 | 15.57 | 0.539 | 0.541 |
| test | 2 | 16.36 | 16.33 | 0.545 | 0.544 |
| test | 4 | 17.20 | 17.33 | 0.544 | 0.544 |
Paired comparison, random S minus fixed S = 2: per-sample mean difference over the 4 common target views {1, 3, 4, 6} of each of the 28 samples; 95% interval from 5000 bootstrap resamples of the samples. Wins = samples where random S is higher.
| Split | S | ΔPSNR (dB) [95% CI] | Wins / 28 | ΔSSIM [95% CI] | Wins / 28 |
|---|---|---|---|---|---|
| val | 1 | -0.00 [-0.05, +0.05] | 13 | +0.0026 [+0.0009, +0.0042] | 20 |
| val | 2 | -0.02 [-0.06, +0.03] | 12 | +0.0019 [+0.0001, +0.0037] | 16 |
| val | 4 | +0.11 [+0.04, +0.20] | 17 | +0.0006 [-0.0015, +0.0027] | 12 |
| test | 1 | -0.01 [-0.06, +0.04] | 17 | +0.0027 [+0.0009, +0.0044] | 21 |
| test | 2 | -0.01 [-0.06, +0.04] | 14 | +0.0003 [-0.0014, +0.0020] | 16 |
| test | 4 | +0.13 [+0.06, +0.20] | 22 | -0.0008 [-0.0028, +0.0013] | 12 |
Validation predictions, correct conditioning. One figure for each S. In each figure, the source views and ground truth are the same for both models. Columns: source views, ground truth, then the prediction of the fixed S = 2 model and of the random S model. Rows: target views.
S = 1
S = 2
S = 4
The model trained with a random number of source views (S drawn from {1, 2, 3, 4}) performs similarly to the model trained with S = 2. Random S is slightly better in some settings (see Limitations).
With random S, the mean number of source views during training is 2.5, a little higher than 2. The Q2.2 ablation shows that more source views at test time improve the result. This is intuitive: more source views cover a larger part of the space, so a larger part of the target view overlaps a source view and is grounded by it. A higher mean S during training can help in a similar way.
The higher mean S during training does not improve PSNR and SSIM at every test-time S. On the shared target views, random S gives a higher PSNR only at S = 4, and a higher SSIM only at S = 1 and at S = 2 on validation. The other differences are inside the 95% intervals. The improvements are too small to see in the images.
python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit: one fixed training clip, 8 views, 40 epochs × 25 steps, AdamW, learning rate 1e-3 with cosine decay to 5%, no warmup. Loss = 5 × camera + 1 × depth.
Ground truth (left) and prediction (right), same clip, canonical first-camera coordinates. Red frustum: view 0. View from the ceiling, orthographic: the viewer camera looks straight down the gravity axis, estimated as the normal of the largest horizontal plane in the GT points (floor or table tops). The first camera looks up the image. Legend and title hidden, points drawn larger.
configs/vggt.json.num_views=8, sampling_range=32), batch size 16.best.pt = lowest validation loss (epoch 969).
Evaluation of best.pt: 8 views from the first 32 prepared frames, 224 px, seed 825. 28 samples per split.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel (lower is better) | 0.133 | 0.156 |
| Depth δ1 | 0.848 | 0.830 |
| Rotation AUC@5 | 0.248 | 0.260 |
| Rotation AUC@15 | 0.581 | 0.609 |
| Rotation AUC@30 | 0.742 | 0.778 |
| Translation AUC@5 | 0.037 | 0.015 |
| Translation AUC@15 | 0.104 | 0.090 |
| Translation AUC@30 | 0.209 | 0.221 |
| Joint pose AUC@5 | 0.014 | 0.004 |
| Joint pose AUC@15 | 0.082 | 0.071 |
| Joint pose AUC@30 | 0.185 | 0.204 |
Left: ground truth. Right: prediction (predicted cameras and depth, unprojected into the first-camera frame). Same view from the ceiling as in Q3.1.
Scene 09c1414f1b
Scene 21d970d8de
Scene 280b83fcf3
Scene 2b1dc6d6a5
Depth of view 0 for the four validation scenes: GT, prediction and per-pixel AbsRel. Magenta: pixels without valid GT depth (not scored).
Predicted (orange) and GT (gray) cameras of the same four samples in the first-camera frame, without alignment (view 0 is the identity in both). Top: top view (X–Z). Bottom: side view (Z–Y). Dotted lines join the GT and predicted centers of the same view. The text gives the rotation error of views 1–7.
Depth. The largest depth errors are near large depth discontinuities. Near an edge, the model tends to average the depths of the two surfaces, so in the predicted depth maps the edges are blurred and show a gradient instead of a step. In the point clouds, this puts points between the foreground and the background surface.
Small camera baseline. The cameras of one sample are close together: the 8 views come from a range of 32 consecutive frames. A new view therefore adds little parallax, and the depth does not get much correction from the other views.
Camera pose. Most of the camera pose error comes from translation (translation AUC is much lower than rotation AUC). This can be related to the limited depth understanding of the model.
Is a point map from a dedicated point head better than points from the predicted depth and cameras? Does the extra point head change the camera and depth results?
configs/vggt_points.json): the baseline plus a point head. The point head is a linear layer to 3 × 14 × 14 values per patch, then the same RGB refinement as the depth head, no activation. It predicts world points in the first-camera frame.scripts/vggt_point_compare.py): L2 distance to the GT points on valid pixels, in normalized units and in meters; Inlier@t = share of valid pixels with L2 < t. "Unprojected" = points from predicted depth and cameras. "Direct" = points from the point head.| Split | Points | L2 (normalized) | L2 (m) | Inlier@0.05 | Inlier@0.1 | Inlier@0.2 |
|---|---|---|---|---|---|---|
| val | Baseline, unprojected | 0.239 | 0.483 | 0.133 | 0.333 | 0.631 |
| val | Point-head model, unprojected | 0.226 | 0.460 | 0.134 | 0.354 | 0.640 |
| val | Point-head model, direct | 0.227 | 0.464 | 0.115 | 0.337 | 0.638 |
| test | Baseline, unprojected | 0.215 | 0.405 | 0.116 | 0.322 | 0.652 |
| test | Point-head model, unprojected | 0.177 | 0.342 | 0.149 | 0.414 | 0.746 |
| test | Point-head model, direct | 0.188 | 0.360 | 0.107 | 0.376 | 0.726 |
Camera and depth metrics of best.pt (point-head model: epoch 998).
| Metric | Baseline val | Point head val | Baseline test | Point head test |
|---|---|---|---|---|
| Depth AbsRel (lower is better) | 0.133 | 0.134 | 0.156 | 0.127 |
| Depth δ1 | 0.848 | 0.852 | 0.830 | 0.859 |
| Rotation AUC@5 | 0.248 | 0.258 | 0.260 | 0.262 |
| Rotation AUC@15 | 0.581 | 0.589 | 0.609 | 0.597 |
| Rotation AUC@30 | 0.742 | 0.749 | 0.778 | 0.775 |
| Translation AUC@5 | 0.037 | 0.023 | 0.015 | 0.018 |
| Translation AUC@15 | 0.104 | 0.100 | 0.090 | 0.116 |
| Translation AUC@30 | 0.209 | 0.225 | 0.221 | 0.247 |
| Joint pose AUC@5 | 0.014 | 0.011 | 0.004 | 0.005 |
| Joint pose AUC@15 | 0.082 | 0.087 | 0.071 | 0.093 |
| Joint pose AUC@30 | 0.185 | 0.207 | 0.204 | 0.230 |
Point-head model loss curves.
Four validation scenes, same view from the ceiling as in Q3.2. Left to right: ground truth, point-head model with points from predicted depth and cameras, point-head model with points from the point head.
Scene 09c1414f1b
Scene 21d970d8de
Scene 280b83fcf3
Scene 2b1dc6d6a5
The point head improves the depth result and, on the test split, the camera result. On validation the changes are small and mixed.
In the point clouds of the four validation scenes, the point-head model puts the points that are far from the camera closer to the GT. The point error binned by GT depth (scripts/vggt_point_error_by_depth.py) agrees on the validation split: the error is mostly lower beyond 2 m and almost the same nearer than 2 m. On the test split the gain is at the near points (nearer than 2 m), and points beyond 3 m have the same or a slightly larger error.
The point head adds a second supervision signal for the same geometry. Its targets are world points in the first-camera frame, which depend on the depth and the camera pose together, so its gradients bring information about both into the shared backbone. This extra multi-task supervision can make the shared features better for the depth and camera heads.
The point head does not fix the depth averaging at depth discontinuities: the point clouds of the point-head model still put many points between the foreground and the background surface. Camera translation also stays a problem: the translation AUC of the point-head model is still much lower than its rotation AUC.
How does the number of input views change accuracy, runtime and memory? Does training with a random number of views change the results?
configs/vggt_random_views.json): at each step, draw k uniformly from 2 to 8 and keep the first k of the 8 loaded views (view 0 stays the reference camera). Poses, depth and points are rescaled so that the k-view valid-point centroid is at distance 1. All other settings are the same as the baseline. Validation during training uses 8 views.best.pt files at V = 2, 4, 8 views from the first 32 prepared frames, seed 825 (view pairs per split: 28, 168, 784). 95% intervals from 2000 bootstrap resamples of the 28 samples.scripts/vggt_view_benchmark.py).| Split | V | Model | Rot AUC@30 [95% CI] | Trans AUC@30 [95% CI] | AbsRel [95% CI] | δ1 [95% CI] | Joint AUC@30 |
|---|---|---|---|---|---|---|---|
| val | 2 | Baseline | 0.768 [0.654, 0.861] | 0.238 [0.126, 0.351] | 0.128 [0.095, 0.178] | 0.860 [0.773, 0.920] | 0.236 |
| val | 2 | Random views | 0.723 [0.621, 0.814] | 0.236 [0.125, 0.358] | 0.111 [0.089, 0.137] | 0.864 [0.785, 0.924] | 0.201 |
| val | 4 | Baseline | 0.725 [0.641, 0.796] | 0.220 [0.155, 0.288] | 0.133 [0.095, 0.184] | 0.831 [0.725, 0.914] | 0.191 |
| val | 4 | Random views | 0.699 [0.622, 0.769] | 0.201 [0.141, 0.269] | 0.130 [0.101, 0.171] | 0.861 [0.784, 0.917] | 0.183 |
| val | 8 | Baseline | 0.742 [0.663, 0.814] | 0.209 [0.154, 0.272] | 0.133 [0.098, 0.186] | 0.848 [0.765, 0.911] | 0.185 |
| val | 8 | Random views | 0.712 [0.637, 0.774] | 0.183 [0.132, 0.246] | 0.146 [0.109, 0.198] | 0.824 [0.733, 0.897] | 0.169 |
| test | 2 | Baseline | 0.799 [0.708, 0.871] | 0.181 [0.080, 0.290] | 0.106 [0.090, 0.122] | 0.881 [0.835, 0.919] | 0.168 |
| test | 2 | Random views | 0.729 [0.627, 0.824] | 0.230 [0.119, 0.345] | 0.111 [0.090, 0.134] | 0.884 [0.829, 0.931] | 0.198 |
| test | 4 | Baseline | 0.786 [0.727, 0.834] | 0.222 [0.153, 0.290] | 0.139 [0.103, 0.193] | 0.831 [0.756, 0.891] | 0.202 |
| test | 4 | Random views | 0.776 [0.713, 0.826] | 0.191 [0.127, 0.262] | 0.133 [0.100, 0.182] | 0.845 [0.756, 0.909] | 0.168 |
| test | 8 | Baseline | 0.778 [0.719, 0.826] | 0.221 [0.165, 0.276] | 0.156 [0.112, 0.230] | 0.830 [0.744, 0.896] | 0.204 |
| test | 8 | Random views | 0.747 [0.688, 0.799] | 0.183 [0.131, 0.239] | 0.147 [0.106, 0.204] | 0.818 [0.732, 0.887] | 0.169 |
Same evaluated outputs for every V. In the table above, depth is scored over views 0 to V − 1 and pose over 1, 6 or 28 pairs per sample. The view selection is nested, so views 0 and 1 are the same frames for every V. The table below scores only the depth of views 0 and 1 and the pose of pair (0, 1) (one pair per sample). The scene scale still uses all V input views. 95% intervals from 2000 bootstrap resamples of the 28 samples.
| Split | V | Model | Rot AUC@30, pair (0, 1) | Trans AUC@30, pair (0, 1) | AbsRel, views 0–1 | δ1, views 0–1 |
|---|---|---|---|---|---|---|
| val | 2 | Baseline | 0.768 [0.659, 0.858] | 0.238 [0.132, 0.355] | 0.128 [0.095, 0.176] | 0.860 [0.776, 0.921] |
| val | 2 | Random views | 0.723 [0.614, 0.818] | 0.236 [0.125, 0.348] | 0.111 [0.088, 0.136] | 0.864 [0.788, 0.929] |
| val | 4 | Baseline | 0.773 [0.660, 0.862] | 0.320 [0.206, 0.437] | 0.131 [0.093, 0.183] | 0.840 [0.729, 0.919] |
| val | 4 | Random views | 0.740 [0.638, 0.830] | 0.255 [0.144, 0.364] | 0.126 [0.097, 0.169] | 0.871 [0.790, 0.927] |
| val | 8 | Baseline | 0.754 [0.638, 0.858] | 0.333 [0.213, 0.448] | 0.124 [0.088, 0.176] | 0.874 [0.789, 0.934] |
| val | 8 | Random views | 0.721 [0.618, 0.812] | 0.235 [0.130, 0.356] | 0.136 [0.099, 0.193] | 0.842 [0.742, 0.916] |
| test | 2 | Baseline | 0.799 [0.715, 0.873] | 0.181 [0.081, 0.289] | 0.106 [0.090, 0.122] | 0.881 [0.836, 0.921] |
| test | 2 | Random views | 0.729 [0.624, 0.824] | 0.230 [0.124, 0.342] | 0.111 [0.090, 0.135] | 0.884 [0.827, 0.933] |
| test | 4 | Baseline | 0.805 [0.717, 0.873] | 0.236 [0.131, 0.352] | 0.128 [0.098, 0.170] | 0.853 [0.779, 0.909] |
| test | 4 | Random views | 0.788 [0.698, 0.869] | 0.227 [0.111, 0.348] | 0.126 [0.096, 0.160] | 0.846 [0.767, 0.915] |
| test | 8 | Baseline | 0.800 [0.715, 0.871] | 0.270 [0.156, 0.392] | 0.145 [0.107, 0.203] | 0.830 [0.734, 0.900] |
| test | 8 | Random views | 0.763 [0.671, 0.840] | 0.221 [0.120, 0.344] | 0.139 [0.107, 0.184] | 0.835 [0.744, 0.903] |
| Views V | Forward time (ms) | Time per view (ms) | Peak memory (MiB) |
|---|---|---|---|
| 2 | 10.6 | 5.3 | 709 |
| 4 | 10.5 | 2.6 | 727 |
| 8 | 13.4 | 1.7 | 775 |
| 16 | 20.8 | 1.3 | 856 |
| 32 | 40.3 | 1.3 | 1016 |
Random-view model loss curves.
Four validation scenes at 8 views, same view from the ceiling as in Q3.2. Left to right: ground truth, baseline, random-view model.
Scene 09c1414f1b
Scene 21d970d8de
Scene 280b83fcf3
Scene 2b1dc6d6a5
The random-view model is slightly worse than the baseline, mostly in camera pose: its rotation AUC@30 is lower at every V on both splits, and its translation AUC@30 is lower at V = 4 and 8. A possible reason is that the mean number of views during training decreases: k is drawn uniformly from 2 to 8, so the model sees 5 views per sample on average instead of 8. The depth results are mixed, and all differences are inside the 95% intervals.