Compute. Part 1, Q2.1 and Q3.1 ran locally on an Apple M4 Max (Q2.1 on MPS in fp32, Q3.1 on CPU because the provided VGGT metrics use float64). Full training for Q2.2, Q2.3 and Q3.2 and the timing/memory runs of Q3.3 ran on a single NVIDIA A100 40 GB in Google Colab with bf16 autocast. The Q3.3 window experiment ran locally on CPU in fp32. Apart from the TODOs, the only code changes are a per-step sampled source count in the LVSM training loop for Q2.3 and a separate evaluation script for Q3.3; the provided training, evaluation and metric code is otherwise unchanged.
The same 8 frames of each scene are shown in the original world frame (meters, after Q1.1–1.2) and after canonicalization (Q1.3). In the canonicalized view the first camera (red frustum) sits at the origin with identity pose, and the side panel reports a valid-point centroid distance of 1.000 normalized units.
In both stages the point clouds from different views overlap without visible double surfaces, which indicates that intrinsics, z-depth unprojection and camera-to-world poses are consistent.
Output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
(full text: check_data_output.txt):
| Sample | Views | Resolution | Valid depth | Max point reproj. error (px) | Max ray reproj. error (px) | Centroid distance |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 97.8% | 1.11e-4 | 9.65e-5 | 1.0000 |
| 5f99900f09_frame_002820 | 8 | 224 | 99.0% | 1.11e-4 | 1.08e-4 | 1.0000 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 96.1% | 1.07e-4 | 8.45e-5 | 1.0000 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 75.2% | 1.14e-4 | 9.86e-5 | 1.0000 |
batch shapes (batch size 2): rgb, ray_origins, ray_directions, points_world (2, 8, 3, 224, 224) depth, depth_valid (2, 8, 1, 224, 224) plucker (2, 8, 6, 224, 224) K (2, 8, 3, 3) T_world_from_camera (2, 8, 4, 4) original_world_from_reference (2, 4, 4) frame_numbers (2, 8) scene_scale (2,)
All reprojection errors are about 1e-4 px, far below the 0.01 px threshold, and every centroid distance is 1 up to float32 rounding.
Every sample is expressed in the coordinate frame of its first sampled camera. With camera-to-world
poses Ti, the new poses are T0−1 Ti, so
camera 0 becomes the identity (computed in float64 and cast back to float32). Depth is unprojected in this
frame, and the scene scale s is the distance from camera 0 to the centroid of all valid points of
the selected views. Translations, depth and points are divided by s; rotations are unchanged
and the centroid is not subtracted, so camera 0 stays at the origin.
The original world frame of a ScanNet++ scan is arbitrary and its metric size varies from room to room, while the images alone determine geometry only up to a rigid transform and a global scale. Canonicalization removes exactly these unobservable degrees of freedom, so the regression targets are consistent across samples and the model never has to guess where the dataset's world origin was.
For pixel center (u, v), the camera-space direction is K−1[u, v, 1]T.
It is rotated into the canonical world frame with the camera-to-world rotation and normalized to unit
length; the ray origin is the camera center t, shared by every pixel of a view:
The intrinsics come from Q1.1. Because the provided resize uses align_corners=False, the
mapping is xout + 0.5 = (sizeout/sizein)(xin + 0.5),
so the focal lengths are scaled by the size ratio and the principal point becomes
c' = r (c + 0.5) − 0.5. Since t is in canonical units, the rays live in the
same frame and scale as the depth targets.
Each ray is stored as six channels [d, m] with moment m = o × d. The moment
does not depend on which point of the line is used as the origin, because
(o + λd) × d = o × d, so (d, m) describes the line itself.
m is perpendicular to the plane that contains the line and the world origin, and since
d is a unit vector, ‖m‖ equals the distance from the origin to the line.
The direction alone tells the model where a pixel looks; the moment adds where the ray is located, which
distinguishes parallel rays from different camera centers. Every ray of camera 0 passes through the
origin, so its moment is zero.
One fixed batch (batch size 2, 2 source views, 6 targets per sample) was fitted for 2000 steps (about 34 minutes on MPS). The final overfit validation PSNR is 34.04 dB (SSIM 0.915), above the 30 dB target.
val_epoch000.png after overfitting. Columns: source views, ground truth,
prediction.configs/lvsm.json (14.6M parameters; patch size 8,
width 384, 8 layers, head dim 64, MLP ratio 4, QK norm).best.pt is epoch 90 (val PSNR 16.07, SSIM 0.558).
log.jsonl.Train and validation PSNR stay close throughout (train 15.46 vs val 16.07 at epoch 90), so the model is not overfitting; the validation curve flattens after roughly epoch 75.
| Source views S | Conditioning | PSNR (dB) | SSIM | Target images |
|---|---|---|---|---|
| 1 | correct | 15.19 | 0.556 | 196 |
| 1 | shuffled | 11.75 | 0.485 | 196 |
| 1 | none | 12.51 | 0.529 | 196 |
| 2 | correct | 16.07 | 0.558 | 168 |
| 2 | shuffled | 11.40 | 0.470 | 168 |
| 2 | none | 12.51 | 0.526 | 168 |
| 4 | correct | 16.78 | 0.574 | 112 |
| 4 | shuffled | 11.55 | 0.475 | 112 |
| 4 | none | 12.52 | 0.535 | 112 |
Columns in every LVSM visualization: source views, ground truth, prediction.
All nine settings: S1 correct / shuffled / none; S2 correct / shuffled / none; S4 correct / shuffled / none.
Number of source views. With correct rays, PSNR rises from 15.19 (S = 1) to 16.07 (S = 2) and 16.78 (S = 4), even though the model was trained only with S = 2. More sources cover more of the room, so a larger fraction of each target can be copied from a nearby view instead of being hallucinated. The gain from 2 to 4 is smaller than from 1 to 2, and SSIM changes much less than PSNR (0.556 to 0.574): the extra views mostly fix colors and coarse layout, while fine structure stays blurry.
Ray conditioning. Removing the target rays (none) drops PSNR to about 12.5 dB for every S. Without target-view conditioning, the predictions become a much blurrier, nearly constant image close to the scene's average colors. They still depend on the source RGB and source rays, so changing S can still affect the output, even though the PSNR here barely moves (12.51 to 12.52 dB). Shuffled rays are worse than no rays (11.4 to 11.8 dB) because the model confidently renders a plausible but wrong viewpoint, and a sharp wrong image is penalized more by MSE than a blurry average. The large gap between correct and shuffled (3.4 to 5.2 dB) shows that the model really uses the target Plücker rays.
Failure modes. Even with correct rays the predictions are blurry and show visible 8×8 patch blocks, because each patch is decoded independently by a linear head and the MSE loss favors averaging when the target content is uncertain. Regions that no source view observes are filled with smooth color.
The baseline is always trained with S = 2 but is evaluated with S = 1, 2 and 4. Does training with a random number of source views per step, S ∈ {1, 2, 3, 4}, make the model better at the S values it was not trained on, and does it cost anything at S = 2?
configs/lvsm_random_s.json is identical to configs/lvsm.json except for
train_source_views: [1, 2, 3, 4]. In train.py, one S is drawn uniformly per batch;
the remaining 8 − S views are targets.best.pt selection still use S = 2, the same as the baseline.| S | Conditioning | Baseline PSNR | Random-S PSNR | ΔPSNR (paired) | 95% CI | Baseline SSIM | Random-S SSIM |
|---|---|---|---|---|---|---|---|
| 1 | correct | 15.19 | 15.16 | −0.03 | [−0.08, +0.03] | 0.556 | 0.559 |
| 1 | shuffled | 11.75 | 11.81 | +0.05 | [+0.02, +0.09] | 0.485 | 0.490 |
| 1 | none | 12.51 | 12.42 | −0.09 | [−0.16, −0.02] | 0.529 | 0.534 |
| 2 | correct | 16.07 | 16.07 | +0.00 | [−0.05, +0.06] | 0.558 | 0.559 |
| 2 | shuffled | 11.40 | 11.47 | +0.07 | [+0.04, +0.10] | 0.470 | 0.473 |
| 2 | none | 12.51 | 12.39 | −0.12 | [−0.20, −0.05] | 0.526 | 0.531 |
| 4 | correct | 16.78 | 16.87 | +0.09 | [+0.02, +0.19] | 0.574 | 0.572 |
| 4 | shuffled | 11.55 | 11.53 | −0.02 | [−0.07, +0.03] | 0.475 | 0.470 |
| 4 | none | 12.52 | 12.42 | −0.11 | [−0.20, −0.01] | 0.535 | 0.541 |
Evaluation images for the random-S model: S1 correct · S2 correct · S4 correct · S2 shuffled · S2 none.
Random-S training gives small, mixed changes and no clear benefit. All nine settings differ from the baseline by at most 0.12 dB PSNR and 0.005 SSIM, the S = 2 validation curves overlap for the whole run, and the predictions look the same. Several paired CIs exclude 0, but in both directions: S = 4 with correct rays (+0.09 dB, CI [+0.02, +0.19]) and shuffled rays at S = 1 and 2 (+0.05 and +0.07 dB) are slightly better, while none is about 0.1 dB worse at every S. At S = 1 with correct rays, the random-S model, which saw S = 1 in a quarter of its steps, is not better (−0.03 dB, CI includes 0). The none rows cannot serve as a pure noise reference: without target rays the source RGB, the source rays and the number of tokens still change with S, and the two models were trained differently. Each configuration was trained once with one seed, so the CIs only cover variation over scenes and images, not over retraining, and I cannot tell whether these small differences would survive another training run.
A likely explanation is that the baseline already generalizes across S. Each target token attends to all source tokens with full attention, so adding or removing a source view only changes the length of the attention context, and the baseline's PSNR at S = 1 and 4 already follows the expected trend. With this small model and a 100-epoch budget, capacity and blur seem to limit quality more than the training view count does.
Limitations. One seed per configuration; best.pt is selected at S = 2 only;
the sampled S range stops at 4, so S > 4 was not tested; and a longer schedule might separate the runs.
The default model (frozen DINOv2 ViT-B/14, 8 frame/global attention pairs, width 384, 6 heads) was fitted to one fixed training clip of 8 views for 40 epochs of 25 steps (1000 steps, lr 1e-3, about 19 minutes on CPU).
| Total loss | Rotation err. (°) | Translation err. (°) | AbsRel | δ1 | Pose AUC@5 | |
|---|---|---|---|---|---|---|
| Before training | 5.49 | 49.7 | 90.0 | 0.619 | 0.208 | 0.00 |
| After 1000 steps | −0.19 | 0.12 | 0.17 | 0.036 | 0.955 | 1.00 |
The loss becomes negative because the confidence term contains −0.2·log(confidence), which keeps decreasing as the model grows more confident on a clip it has memorized.
The predicted cameras land on the GT cameras and the reconstructed room matches the GT point cloud. The predicted depth is slightly blocky at the 14×14 patch scale, since the coarse depth is decoded per patch before RGB refinement.
configs/vggt.json: frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads,
camera head with 4 cross-view blocks, depth head with RGB refinement, no point head.num_views = 8, sampling_range = 32, 224×224, batch size 16
(14 steps per epoch over the 224 training scenes).best.pt is epoch 985 (val loss 0.819). Evaluation uses the required protocol: 8 views from the
first 32 prepared frames, 224px, seed 825, full splits.
The validation loss is below the training loss for the whole run (0.82 vs 0.95 at epoch 985). Training batches use a random 32-frame window and random view order, while validation uses fixed views that always start at frame 0, so the two losses are not measured on the same distribution; I did not run an experiment to confirm which difference causes the gap.
| Split | AbsRel ↓ | δ1 ↑ | Rot AUC @5/15/30 | Trans AUC @5/15/30 | Pose AUC @5/15/30 | Mean rot err. | Mean trans err. |
|---|---|---|---|---|---|---|---|
| Val | 0.134 | 0.840 | 0.178 / 0.499 / 0.696 | 0.019 / 0.083 / 0.196 | 0.005 / 0.064 / 0.175 | 12.3° | 41.1° |
| Test | 0.152 | 0.811 | 0.198 / 0.529 / 0.723 | 0.013 / 0.073 / 0.180 | 0.003 / 0.055 / 0.163 | 9.6° | 39.3° |
28 scenes and 784 view pairs per split, no degenerate pairs. Raw reports: val metrics.json · test_metrics.json.
In each panel: GT cameras and points (top left), points reconstructed from predicted cameras and depth (top right), and the view-0 RGB, GT depth and predicted depth (bottom). The first camera is red.
How does the number of input views V affect accuracy, runtime and memory at evaluation time? In particular, do more views help the weak translation estimates from Q3.2? A follow-up experiment asks whether the result is driven by the number of views or by how far apart the views are.
best.pt, trained only with V = 8. Nothing is retrained; only the input
changes. Evaluation script: models/vggt/eval_views.py.a = median(GT) / median(pred) over its valid pixels before computing AbsRel. This roughly separates
errors in global scale from errors in depth shape. It uses GT, so it is a diagnostic, not a fair metric.max_memory_allocated after
reset_peak_memory_stats, also as the extra peak above the resident model and input.
A100 40 GB, bf16 autocast. No V ran out of memory.sampling_range = 32, so W = 32 matches training. Run on CPU in fp32; the W = 32 result reproduces
the Q3.2 metrics.
| Split | V | Group | Pairs | AbsRel | AbsRel aligned | δ1 | Rot AUC@30 | Trans AUC@30 | Pose AUC@30 | Forward (ms) | Peak (MiB) | Extra (MiB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| val | 2 | all | 28 | 0.122 | 0.085 | 0.853 | 0.710 | 0.185 | 0.176 | 42.3 | 588 | 19 |
| val | 4 | all = first 4 | 168 | 0.127 | 0.089 | 0.846 | 0.663 | 0.158 | 0.144 | 42.8 | 605 | 35 |
| val | 8 | all | 784 | 0.134 | 0.093 | 0.840 | 0.697 | 0.195 | 0.175 | 42.5 | 648 | 77 |
| first 4 | 168 | 0.136 | 0.088 | 0.832 | 0.681 | 0.178 | 0.152 | |||||
| val | 16 | all | 3360 | 0.166 | 0.096 | 0.793 | 0.672 | 0.182 | 0.156 | 42.8 | 729 | 154 |
| first 4 | 168 | 0.170 | 0.089 | 0.798 | 0.701 | 0.167 | 0.149 | |||||
| val | 32 | all | 13888 | 0.174 | 0.097 | 0.785 | 0.664 | 0.175 | 0.149 | 44.7 | 888 | 303 |
| first 4 | 168 | 0.178 | 0.090 | 0.792 | 0.682 | 0.164 | 0.141 | |||||
| test | 2 | all | 28 | 0.122 | 0.093 | 0.866 | 0.735 | 0.185 | 0.118 | 42.7 | 588 | 19 |
| test | 4 | all = first 4 | 168 | 0.140 | 0.100 | 0.824 | 0.746 | 0.190 | 0.158 | 42.8 | 605 | 35 |
| test | 8 | all | 784 | 0.152 | 0.097 | 0.811 | 0.723 | 0.181 | 0.163 | 43.1 | 648 | 77 |
| first 4 | 168 | 0.154 | 0.099 | 0.806 | 0.738 | 0.210 | 0.182 | |||||
| test | 16 | all | 3360 | 0.162 | 0.095 | 0.792 | 0.726 | 0.183 | 0.167 | 43.5 | 729 | 154 |
| first 4 | 168 | 0.164 | 0.098 | 0.787 | 0.718 | 0.196 | 0.167 | |||||
| test | 32 | all | 13888 | 0.168 | 0.095 | 0.790 | 0.744 | 0.179 | 0.165 | 43.8 | 888 | 303 |
| first 4 | 168 | 0.170 | 0.099 | 0.790 | 0.719 | 0.201 | 0.172 |
The V = 8 all-views row matches the official Q3.2 evaluation (AbsRel and δ1 exactly, AUC within 0.001), which checks the script. Full tables: val csv · test csv.
F.scaled_dot_product_attention), but I did not check which kernel was used.
Linear memory does not mean linear compute: the FLOPs of global attention still grow quadratically with the
total token count. Total peak memory goes from 0.57 to 0.87 GiB, well within the GPU.In Experiment A the window is fixed at 32 frames, so a larger V also means views that are closer together. To test whether translation fails because neighboring views have too little parallax, Experiment B keeps V = 8 and changes how far apart the views are. Changing W also changes which frames are sampled, so the images, the scene content they cover and the camera pairs being scored all change together; parallax is not the only variable. The table also lists the median GT distance between camera centers over all pairs and the median relative rotation.
| Split | W | Median baseline (m) | Median rel. rotation | AbsRel | AbsRel aligned | Rot AUC@30 | Trans AUC@30 | Δ Trans AUC@30 vs W = 32 [95% CI] | Pose AUC@30 |
|---|---|---|---|---|---|---|---|---|---|
| val | 16 | 0.14 | 16.8° | 0.111 | 0.088 | 0.782 | 0.123 | −0.072 [−0.122, −0.025] | 0.114 |
| val | 32 | 0.29 | 26.7° | 0.134 | 0.093 | 0.697 | 0.194 | reference | 0.175 |
| val | 64 | 0.44 | 41.2° | 0.222 | 0.105 | 0.508 | 0.188 | −0.006 [−0.048, +0.033] | 0.129 |
| val | 128 | 0.81 | 57.1° | 0.377 | 0.121 | 0.352 | 0.121 | −0.073 [−0.119, −0.028] | 0.061 |
| test | 16 | 0.10 | 11.3° | 0.113 | 0.089 | 0.840 | 0.112 | −0.070 [−0.118, −0.023] | 0.102 |
| test | 32 | 0.22 | 23.3° | 0.152 | 0.097 | 0.723 | 0.181 | reference | 0.163 |
| test | 64 | 0.39 | 32.7° | 0.228 | 0.104 | 0.615 | 0.155 | −0.027 [−0.070, +0.015] | 0.127 |
| test | 128 | 0.63 | 54.8° | 0.325 | 0.116 | 0.384 | 0.160 | −0.021 [−0.065, +0.021] | 0.100 |
The CI is a scene-level bootstrap of the paired per-scene difference (every scene has the same 28 pairs). Full table: window_summary.csv.
For this model, the input that works best is the one it was trained on: 8 views spread over a 32-frame window. Adding views at test time keeps the scale-aligned depth error and the camera accuracy roughly unchanged, but makes the absolute scale less reliable, so raw AbsRel gets worse. It costs very little: runtime is flat and memory grows linearly up to 32 views. Translation, the main weakness from Q3.2, was not improved by either test-time change I tried. More views do not help, a narrower window makes it worse, and a wider window does not help and moves the input out of the training distribution. Improving it likely requires changes during training, for example training with more and wider views or a stronger camera head.
sampling_range = 32. A model trained with varying V
or wider windows could behave differently; that needs retraining (about 3.6 hours per run).