Liza Dahiya · ldahiya@andrew.cmu.edu
All Q1 functions are implemented in a4data/geometry.py. The figures use the same five training samples
as scripts/visualize_data.py (seed 825, 8 views, 224×224). The red frustum is camera 0 and blue
frustums are the other views. Points are colored by RGB and subsampled for display.
After unprojection, points and cameras are in the original ScanNet++ world frame, in meters. Camera 0 sits at an arbitrary position, and the centroid of the valid points is several meters from the origin. After canonicalization, camera 0 is the identity pose at the origin, and the valid-point centroid is exactly 1 unit away.
| Sample | Camera 0 center, world (m) | Centroid distance (m) | Camera 0 center, canonical | Centroid distance, canonical |
|---|---|---|---|---|
036bce3393_frame_004470 | (3.346, 4.677, 1.468) | 6.668 | (0.000, 0.000, 0.000) | 1.0000 |
076c822ecc_frame_002890 | (1.540, 2.767, 1.586) | 4.435 | (0.000, 0.000, 0.000) | 1.0000 |
079a326597_frame_007430 | (3.756, 2.082, 1.532) | 2.718 | (0.000, 0.000, 0.000) | 1.0000 |
07f5b601ee_frame_003630 | (3.094, 3.489, 1.408) | 5.118 | (0.000, 0.000, 0.000) | 1.0000 |
07ff1c45bb_frame_000460 | (1.340, 2.171, 1.516) | 3.285 | (0.000, 0.000, 0.000) | 1.0000 |
The metric centroid distance (third column) is the scene_scale that divides all translations,
depths and points.
Screenshots of outputs/unprojection/index.html (left) and outputs/canonicalization/index.html (right) for the same five samples. The side-panel note under the depth map gives the first camera center and the valid-point centroid distance. After canonicalization these read (0.0000, 0.0000, 0.0000) and 1.000 normalized units for every scene.
036bce3393_frame_004470076c822ecc_frame_002890079a326597_frame_00743007f5b601ee_frame_00363007ff1c45bb_frame_000460036bce3393_frame_004470
076c822ecc_frame_002890
079a326597_frame_007430
07f5b601ee_frame_003630
07ff1c45bb_frame_000460
check_data.pypython scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
| Sample | Views | Res. | Valid depth | Max point reproj. err. (px) | Max ray reproj. err. (px) | Centroid distance |
|---|---|---|---|---|---|---|
e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 1.10e-04 | 1.08e-04 | 1.000000 |
5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.23e-04 | 1.11e-04 | 1.000000 |
47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 9.88e-05 | 8.90e-05 | 1.000000 |
4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.13e-04 | 1.02e-04 | 1.000000 |
Every reprojection error is about 1e-4 px, well under the 0.01 px requirement, and every centroid distance is 1.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011280165199423209,
"max_ray_reprojection_error_px": 0.00010225349979009479,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization. Absolute ScanNet++ coordinates are arbitrary: each scan has its own origin, orientation and size. To give the network a consistent input distribution, each sample is expressed relative to its first sampled camera. With camera 0's pose $T_{w\leftarrow c_0}$, every pose becomes $T'_i = T_{w\leftarrow c_0}^{-1}\,T_{w\leftarrow c_i}$, computed in float64, so camera 0 is the identity. Depth is unprojected in this frame. The scene scale $s = \lVert \operatorname{mean}(p_\text{valid}) \rVert$ is the distance from camera 0 to the centroid of the valid points pooled over all views. Translations, depths and points are divided by $s$. Rotations, and therefore ray directions, are unchanged. The centroid is not subtracted, so camera 0 stays at the origin. This removes the global rigid transform and scale, which cannot be recovered from images alone.
World-space ray directions. Each pixel center $(u,v)$ is lifted to the camera-frame direction $K^{-1}[u,v,1]^\top$. This vector has unit z, so multiplying it by the z-depth gives the 3D point. Rotating it by the camera-to-world rotation and normalizing gives $d = R\,K^{-1}[u,v,1]^\top / \lVert R\,K^{-1}[u,v,1]^\top \rVert$. Every ray of a view starts at that camera's center $o$, the translation of its pose. The rays therefore encode both the intrinsics (how directions spread across pixels) and the extrinsics (the rotation and the center).
Plücker moment. Each ray is stored as $[d, m]$ with moment $m = o \times d$. The moment is perpendicular to the plane that contains the ray and the origin, and $\lVert m \rVert$ is the ray's distance from the origin. Because $(o + t d)\times d = o\times d$, $m$ does not depend on which point of the ray is used as its origin, so $[d,m]$ identifies the line itself. This gives a dense, per-pixel camera encoding that a transformer can take as image channels. For camera 0, $o = 0$, so its moments are zero.
Implemented in models/lvsm/model.py (patchify/unpatchify, build_tokens, forward),
models/lvsm/losses.py (MSE) and models/lvsm/train.py (bfloat16 autocast forward, float32 loss).
Each target view gets its own sequence: the tokens of all source views (RGB in [−1,1] plus Plücker rays),
followed by that target's ray-only query tokens. The model never sees target RGB.
After 2000 steps on one fixed batch (S=2, 6 targets per sample), the model reaches PSNR 31.00 dB (SSIM 0.872) on that batch.

val_epoch000.png. Columns: source 1, source 2, ground truth, prediction.best.pt is from the last epoch (99).
Validation PSNR rises from 12.2 dB to 16.13 dB and is still climbing at epoch 99. Training PSNR (15.6 dB) is no higher than validation PSNR, so the model is under-fitting, not over-fitting. Its quality is limited by the short training budget (5.6k steps). The overfit experiment shows that the implementation itself can reach >30 dB.
python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4
--conditioning correct shuffled none. Validation split, 28 scenes, the first 32 prepared frames, 8 views per sample.
| Source views | Target images | Correct rays | Shuffled rays | No rays (zeros) | |||
|---|---|---|---|---|---|---|---|
| PSNR ↑ | SSIM ↑ | PSNR ↑ | SSIM ↑ | PSNR ↑ | SSIM ↑ | ||
| 1 | 196 | 15.19 | 0.558 | 11.80 | 0.488 | 12.47 | 0.528 |
| 2 | 168 | 16.13 | 0.560 | 11.48 | 0.472 | 12.44 | 0.527 |
| 4 | 112 | 16.87 | 0.575 | 11.59 | 0.473 | 12.45 | 0.537 |
Training-time validation snapshots at the first and last visualized epochs:


Evaluation with correct target rays and 1, 2 and 4 source views:



Ray-conditioning ablation with S=2:


Number of source views. With correct rays, PSNR rises from 15.19 dB (S=1) to 16.13 dB (S=2) and 16.87 dB (S=4), and SSIM improves too. This happens even though the model was trained only with S=2. More sources cover more of the scene, so more target pixels are visible in some source, and the model needs to hallucinate less. Full attention accepts any number of source tokens, so the same weights run at S=1 and S=4 without retraining. Going from 2 to 4 views gains less than going from 1 to 2, probably because views sampled from the same 32-frame window overlap a lot, so extra views add less new content.
Ray conditioning. The target Plücker rays are the only signal telling the model which view to render, and removing or corrupting them causes the largest drop of any setting. With S=2, shuffled rays drop PSNR to 11.48 dB (−4.6 dB) and zero rays to 12.44 dB (−3.7 dB), regardless of the number of sources. Shuffled rays are worse than no rays. A likely explanation is that wrong rays make the model confidently render the wrong viewpoint, while zero rays are out of distribution and push it toward a blurry, scene-average image, which costs less under MSE. So the model is using the target camera, not just copying or averaging the sources.
Limitations. Predictions get the coarse layout and colors right but are blurry and show 8×8 patch artifacts. MSE alone rewards predicting the average of plausible images, which looks blurry, and 5.6k steps is a short schedule. I would expect longer training (validation PSNR had not plateaued) and an added perceptual or SSIM loss to help most, but I did not test either.
Question. The baseline always trains with S=2 source views, yet it is evaluated with 1, 2 and 4. Does training with a random number of source views per step (S uniform in {1,2,3,4}, targets = the remaining views) give a model that generalizes better across source counts, especially to S=1 and S=4, which the baseline never sees during training?
Setup (controlled). Identical to the 2.2 baseline (same model, optimizer, LR schedule, 100 epochs, batch
size 4, data seed 825, 8 views per sample); the only change is that each training step samples S∈{1,..,4}
(new optional train_source_views setting in models/lvsm/train.py; config
configs/lvsm_randS.json). Validation during training and checkpoint selection still use S=2, so both
runs are selected the same way. Both are evaluated with the same 9-setting ablation as 2.2. Training took about
89 min on a shared A100.

| Eval S | Rays | Fixed S=2 (baseline) | Random S∈{1..4} | Δ PSNR | ||
|---|---|---|---|---|---|---|
| PSNR ↑ | SSIM ↑ | PSNR ↑ | SSIM ↑ | |||
| 1 | correct | 15.19 | 0.558 | 15.28 | 0.559 | +0.08 |
| 1 | shuffled | 11.80 | 0.488 | 11.77 | 0.486 | -0.04 |
| 1 | none | 12.47 | 0.528 | 12.50 | 0.526 | +0.03 |
| 2 | correct | 16.13 | 0.560 | 16.23 | 0.560 | +0.10 |
| 2 | shuffled | 11.48 | 0.472 | 11.44 | 0.468 | -0.05 |
| 2 | none | 12.44 | 0.527 | 12.47 | 0.522 | +0.03 |
| 4 | correct | 16.87 | 0.575 | 17.10 | 0.575 | +0.23 |
| 4 | shuffled | 11.59 | 0.473 | 11.47 | 0.465 | -0.12 |
| 4 | none | 12.45 | 0.537 | 12.50 | 0.530 | +0.04 |


Results. With correct rays, the random-S model is better at every source count: +0.08 dB at S=1, +0.10 dB at S=2 and +0.23 dB at S=4. SSIM is essentially unchanged (within ±0.001). The largest gain is at S=4, which is the setting furthest from what the baseline trained on; the random-S model saw 4-source sequences during training and the baseline never did. At S=2 the random-S model is no worse even though only about a quarter of its training steps used S=2. The shuffled and zero-ray settings stay about equally bad for both models, so the random-S model is just as dependent on correct target rays.
Conclusion and limitations. Random-S training did not hurt at S=2 and helped a little at S=4, so it is a cheap change worth keeping. The gains are small, though (at most 0.23 dB), and come from a single training run per configuration. The epoch-to-epoch fluctuation of the validation curve is about 0.1 dB, so the S=1 and S=2 differences are within plausible run-to-run noise; only the S=4 gain stands somewhat above it, and repeated seeds would be needed to confirm it. Both models remain under-trained (blurry predictions, PSNR still rising at epoch 100), and a longer schedule would likely matter more than the view-count curriculum.
Implemented in models/vggt/model.py (alternating frame/global attention; the last frame-block and
last global-block outputs are concatenated to 2D and layer-normed), models/vggt/heads.py (camera MLP
with ReLU on FoV; depth = exp of patch-decoded logits plus an RGB-conditioned conv residual; confidence = 1 + exp)
and models/vggt/losses.py (translation/quaternion/FoV L1; trimmed regression,
confidence-weighted and 4-scale gradient depth losses).
The 3D figures below are static renders I made from the same predictions that the provided HTML
viewer shows (submission/make_vggt_figures.py). The red frustum is camera 0. Predicted points are
unprojected from the predicted depth using the predicted cameras. GT and prediction share axis limits.
One fixed training clip, 40 epochs × 25 steps, LR 0.001. On that clip, the final model reaches depth AbsRel 0.037 and δ1 0.955, with a mean rotation error of 0.27° and a mean translation-direction error of 0.32° (pose AUC@30 0.999). The depth loss goes negative, as expected, because of the −0.2 log(confidence) term.

Viewer screenshots (outputs/vggt_overfit/visualization/index.html), same camera angle, switching
the geometry selector between ground truth and predicted cameras + depth:
036bce3393_frame_004470 (overfit clip)
last.pt with the optimizer and schedule state intact.
best.pt (lowest validation loss) is from epoch 913.
Depth loss converges in the first ~100 epochs. Camera loss keeps decreasing for the whole run, so pose accuracy would likely improve further with more training. Validation loss stays below training loss for the whole run. The two splits are sampled differently (in training, the reference view is a random frame of a random 32-frame window; in validation it is always frame 0 of the prepared clip), and the 28 validation scenes may simply be easier. I did not isolate which of these causes the gap.
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split {val,test}
--eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825.
28 scenes and 784 view pairs per split, with no degenerate pairs.
| Split | AbsRel ↓ | δ1 ↑ | Rotation AUC ↑ | Translation AUC ↑ | Joint pose AUC ↑ | Mean rot. err. | Mean transl. err. | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @5° | @15° | @30° | @5° | @15° | @30° | @5° | @15° | @30° | |||||
| Validation | 0.145 | 0.830 | 0.203 | 0.522 | 0.695 | 0.015 | 0.084 | 0.193 | 0.007 | 0.060 | 0.168 | 13.2° | 41.0° |
| Test | 0.169 | 0.790 | 0.199 | 0.522 | 0.717 | 0.015 | 0.077 | 0.185 | 0.003 | 0.055 | 0.164 | 9.7° | 41.1° |
Viewer screenshots (outputs/vggt_train/visualization/index.html); left ground truth, right
prediction from best.pt. The side-panel note under each prediction gives its valid-point centroid
distance, which differs from 1 by the model's scale error.
09c1414f1b_frame_00000021d970d8de_frame_000000280b83fcf3_frame_0000002b1dc6d6a5_frame_000000Static renders of the same four scenes, with GT and predicted depth maps for all 8 views:




Depth generalizes reasonably: AbsRel is 0.145 on validation and 0.169 on test, and δ1 is 0.83 and 0.79. The coarse scene layout, including floor, walls and large furniture, is correct and consistent across views, so the fused point clouds have the right overall shape. Errors concentrate in the details. Predicted depth is smooth and blocky at the 14-px DINO patch scale, and thin structures such as chair legs disappear; the two-layer 3×3 conv refinement on RGB is too small to restore them. Object boundaries are smeared, so the predicted clouds are noisier near depth discontinuities. Windows and other regions with no valid GT depth receive no supervision.
Rotation is learned moderately well: AUC@30 is 0.70 on validation and 0.72 on test, with a mean error of about 13° and 10°. Fine accuracy is still poor, however (AUC@5 is about 0.20).
Translation direction is the main failure mode: AUC@30 is only 0.19 and 0.19, with mean errors of about 41°, and it caps the joint pose AUC. Part of the reason is short baselines: on the validation clips, the median distance between two cameras is 0.29 m against a median scene depth of 2.05 m, and 21% of view pairs are less than 10 cm apart. For such pairs, a translation error of a few centimeters already changes the direction by tens of degrees. The camera loss was still falling when training ended, so longer training may help; VGGT's iterative camera refinement is another option I did not try. Test depth is slightly worse than validation while test rotation is slightly better, which is consistent with scene-to-scene variation across only 28 scenes per split.
Question. The model was trained with exactly 8 views. There is no view-order embedding (only camera 0 gets its own camera and register tokens), so the same weights accept any number of views. How do accuracy, runtime and memory change when evaluating with 2, 4, 8 or 16 views, and is "more views is better" true for this model?
Setup. The trained best.pt from 3.2, with no retraining, evaluated on the validation split using
the same protocol as the official evaluation (first 32 prepared frames, 224px, seed 825); only num_views
changes. The loader always keeps frame 0 as the reference and draws the other V−1 views at random from frames
1–31. The V=8 row reproduces the official validation numbers exactly. Latency (median of 10 forward passes after
warm-up) and peak activation memory (excluding the weights) are measured for one sample, in bf16, on an A100.
Script: submission/bonus_vggt_views.py. To separate scale errors from shape errors, I also report AbsRel
after per-sample median scale alignment (submission/bonus_vggt_views_scale.py).
| Views | Pose pairs | AbsRel ↓ | AbsRel, scale-aligned ↓ | |log scale err.| ↓ | δ1 ↑ | Rot. AUC@30 ↑ | Transl. AUC@30 ↑ | Pose AUC@30 ↑ | Mean rot. err. | Peak mem. (MiB) | Latency (ms) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 28 | 0.109 | 0.080 | 0.064 | 0.879 | 0.698 | 0.236 | 0.201 | 13.7° | 143 | 71 |
| 4 | 168 | 0.133 | 0.087 | 0.089 | 0.845 | 0.671 | 0.174 | 0.148 | 13.3° | 160 | 79 |
| 8 | 784 | 0.145 | 0.090 | 0.098 | 0.830 | 0.695 | 0.193 | 0.168 | 13.2° | 202 | 106 |
| 16 | 3360 | 0.170 | 0.092 | 0.108 | 0.807 | 0.673 | 0.168 | 0.145 | 13.4° | 279 | 96 |

Results.
Conclusion and limitations. For this small model, adding views does not improve per-view geometry. It mostly makes the scene-normalized scale harder to predict, while relative depth shape and pose stay nearly constant. Depth metrics computed after scale alignment would separate the two effects, and training with a variable view count might let the model benefit from extra views; I did not test the latter for VGGT. Limitations: one checkpoint, one split (28 scenes) and one random view draw per V; small-V pose metrics rest on few pairs; and latency could not be measured cleanly on the shared GPU.