X_cam = Z · K-1[u, v, 1].



{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001155954014393501,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
]
}
batch_shapes:
rgb [2, 8, 3, 224, 224]
depth [2, 8, 1, 224, 224]
depth_valid [2, 8, 1, 224, 224]
K [2, 8, 3, 3]
frame_numbers [2, 8]
T_world_from_camera [2, 8, 4, 4]
original_world_from_reference [2, 4, 4]
scene_scale [2]
points_world [2, 8, 3, 224, 224]
ray_origins [2, 8, 3, 224, 224]
ray_directions [2, 8, 3, 224, 224]
plucker [2, 8, 6, 224, 224]
All reprojection errors are about 1e-4 px (the limit is 0.01 px) and the valid-point centroid distance is 1.0 for every sample.
Canonicalization.
Trel = T0-1 Ti. Camera 0 becomes the identity: at the origin, looking down +Z.World-space ray directions.
K-1[u, v, 1] is the ray direction in camera coordinates.d = normalize(R K-1[u, v, 1]).Plücker moment.
[d, m] with the moment m = o × d.m is perpendicular to the plane that holds the ray and the world origin, and its length is the distance from the origin to the line.(o + t d) × d = o × d.
val_epoch000.png of the overfit run (columns: source views, ground truth, prediction)Training settings. One A100, 100 epochs (5,600 steps), usually 24 s per epoch and 44 min in total. The best checkpoint is from epoch 99.
configs/lvsm.json, resolved){
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": "MSE on target RGB",
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true
},
"num_source_views": 2,
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/scene_splits.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 4,
"num_workers": 4,
"seed": 825
}
}

Validation PSNR / SSIM for each ablation setting. All rows use the same model (trained with S = 2); only the number of source views given at test time and the target-ray conditioning change.
| Source views at test time | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | 15.19 / 0.558 | 11.80 / 0.488 | 12.47 / 0.527 |
| 2 (trained) | 16.13 / 0.560 | 11.49 / 0.472 | 12.44 / 0.526 |
| 4 | 16.88 / 0.575 | 11.59 / 0.473 | 12.45 / 0.537 |
Training visualizations (columns: source views, ground truth, prediction):


Evaluation visualizations, ray-conditioning ablation (S = 2):



Evaluation visualizations, number of source views (correct rays). One model, trained with S = 2, is given 1, 2 or 4 source views at test time:







Number of source views.
Ray conditioning.
Quality.
Question. If the model sees more source views during training (S = 4 instead of 2), how does this change evaluation with 1, 2 and 4 source views?
Settings. The same as the baseline, except 4 source views (so 4 targets per clip). One A100, about 27 s per epoch, 45 min. The best checkpoint is from epoch 95. Both models are evaluated with the same script, scenes and seed.
configs/lvsm_s4.json, resolved){
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": "MSE on target RGB",
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true
},
"num_source_views": 4,
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/scene_splits.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 4,
"num_workers": 4,
"seed": 825
}
}
| Eval S | Conditioning | Trained S=2: PSNR / SSIM | Trained S=4: PSNR / SSIM | Δ PSNR |
|---|---|---|---|---|
| 1 | correct | 15.19 / 0.558 | 15.33 / 0.557 | +0.14 |
| 2 | correct | 16.13 / 0.560 | 16.41 / 0.562 | +0.27 |
| 4 | correct | 16.88 / 0.575 | 17.46 / 0.580 | +0.58 |
| 1 | shuffled | 11.80 / 0.488 | 11.75 / 0.478 | −0.06 |
| 2 | shuffled | 11.49 / 0.472 | 11.36 / 0.460 | −0.13 |
| 4 | shuffled | 11.59 / 0.473 | 11.40 / 0.457 | −0.19 |
| 1 | none | 12.47 / 0.527 | 12.36 / 0.516 | −0.11 |
| 2 | none | 12.44 / 0.526 | 12.33 / 0.512 | −0.12 |
| 4 | none | 12.45 / 0.537 | 12.38 / 0.520 | −0.06 |


Results and conclusions.
Limitations.



Training settings. Default config, two A100s with DDP (global batch 16), 1000 epochs, about 8 s per epoch for train + val and about 3.1 h wall-clock including checkpointing. The best validation loss is at epoch 950.
configs/vggt.json, resolved){
"model": {
"embed_dim": 384,
"num_heads": 6,
"num_layers": 8,
"predict_points": false,
"encoder_name": "dinov2_vitb14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 1000,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"min_lr_ratio": 0.05,
"warmup_epochs": 10
},
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/scene_splits.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 16,
"num_workers": 8,
"seed": 825
},
"run": {
"world_size": 2,
"precision": "bf16"
}
}

Evaluation (best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825):
| Split | Abs-rel ↓ | RMSE ↓ | δ1 ↑ | Rot. err ↓ | Trans. err ↓ | Rot. AUC @5/15/30 | Trans. AUC @5/15/30 | Pose AUC @5/15/30 |
|---|---|---|---|---|---|---|---|---|
| Val | 0.142 | 0.190 | 0.834 | 12.75° | 41.98° | 0.200 / 0.508 / 0.690 | 0.013 / 0.075 / 0.178 | 0.003 / 0.056 / 0.155 |
| Test | 0.155 | 0.199 | 0.801 | 9.51° | 40.53° | 0.191 / 0.535 / 0.723 | 0.009 / 0.074 / 0.184 | 0.001 / 0.057 / 0.166 |
GT vs. predicted geometry on four held-out validation scenes:








Errors on held-out scenes.
Question. How does the number of input views V affect accuracy, runtime and memory for a model trained with V = 8?
Settings.
best.pt and the same 28 validation scenes.scripts/vggt_views_ablation.py.| V | Pose AUC@30 | Rot. AUC@30 | Trans. AUC@30 | Rot. err | Trans. err | Abs-rel | δ1 | Forward (ms) | Peak memory (MB) |
|---|---|---|---|---|---|---|---|---|---|
| 2 | 0.196 | 0.727 | 0.206 | 11.24° | 33.66° | 0.112 | 0.877 | 64.1 | 708 |
| 4 | 0.150 | 0.681 | 0.168 | 12.04° | 41.24° | 0.122 | 0.866 | 56.6 | 725 |
| 8 (trained) | 0.155 | 0.690 | 0.179 | 12.75° | 41.96° | 0.142 | 0.834 | 76.7 | 770 |
| 16 | 0.139 | 0.684 | 0.158 | 13.73° | 44.27° | 0.175 | 0.795 | 90.7 | 848 |
The V = 8 row differs from the Q3.2 val row in the last digit (translation error 41.96° vs 41.98°) because of bf16 nondeterminism and batch size 1 instead of 16.

Results and conclusions.