All training ran on one NVIDIA T4 GPU (15 GB).
Click an image for full size.
| Scene 036bce3393 | Scene 079a326597 |
|---|---|
![]() |
![]() |
In both scenes the first camera center is (0.0000, 0.0000, 0.0000) and the valid-point centroid distance is 1.000. These values are also printed under each plot. Click an image for full size.
| Scene 036bce3393 | Scene 079a326597 |
|---|---|
![]() |
![]() |
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010844774078577757,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.281559323426336e-05,
"max_ray_reprojection_error_px": 8.446346328128129e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010504279634915292,
"max_ray_reprojection_error_px": 9.423537994734943e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization. Left-multiplying every pose by the inverse of the first camera's pose places that camera at the origin with no rotation. Translations, depth and points are then divided by s = ||mean(p_valid)||, how far the average valid point lies from the origin. This gives every scene the same reference frame and scale.
World-space ray directions. For pixel (u, v), K⁻¹[u, v, 1]ᵀ is the camera-space direction. Applying the camera's rotation and dividing by the length gives d, the unit direction in canonical coordinates.
Plücker moment. m = o × d, where o is the camera position. Any point on the ray gives the same m, since (o + td) × d = o × d, so [d, m] describes the line itself. Without m, parallel rays from different cameras would look identical.
Final overfit PSNR: 33.82 dB (SSIM 0.9118), after 2,000 steps at batch size 2.

val_epoch000.png. Columns: source views, ground truth, prediction.
Settings. Batch size 2 and 50 epochs (5,600 steps), 3.5 hours. The config default is batch size 4 and 100 epochs, but batch size 4 did not fit in GPU memory. All other settings are the defaults in configs/lvsm.json, including 2 source views and learning rate 2e-4. The best checkpoint is epoch 46.

Training curves.
| Source views | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | 14.94 / 0.5531 | 11.77 / 0.4903 | 12.45 / 0.5362 |
| 2 | 15.65 / 0.5532 | 11.54 / 0.4798 | 12.40 / 0.5322 |
| 4 | 16.14 / 0.5667 | 11.69 / 0.4858 | 12.42 / 0.5411 |
Columns within each image: source views, ground truth, prediction. Click an image for full size.
| Epoch 0 | Epoch 25 | Epoch 45 |
|---|---|---|
![]() |
![]() |
![]() |
Columns within each image: source views, ground truth, prediction. Click an image for full size.
| Source views | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | ![]() |
![]() |
![]() |
| 2 | ![]() |
![]() |
![]() |
| 4 | ![]() |
![]() |
![]() |
Source views. Given the true rays, PSNR climbs with each added view, from 14.94 dB with one source to 16.14 dB with four, because more of each target is visible in the inputs.
Ray conditioning. The requested viewpoint reaches the model only through the target rays. With no rays, PSNR stays near 12.4 dB at every view count and the output is a flat average color. Shuffled rays score lowest (11.5–11.8 dB) because the model renders the wrong view, which costs more under MSE than an average.
In all settings the predictions are blurry. With correct rays they still get the coarse layout right.

Overfit loss curves. The total can go negative because of the confidence log term.
Click an image for full size.
| Ground truth | Prediction |
|---|---|
![]() |
![]() |
Settings. Batch size 8 and 300 epochs (8,400 steps), 9.9 hours. The config default is batch size 16 and 1,000 epochs, but batch size 16 did not fit in GPU memory and I shortened the run to fit the time available. All other settings are the defaults in configs/vggt.json, including 8 views, sampling range 32 and learning rate 6e-4. Metrics use best.pt from epoch 297.

Normal-training loss curves.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.1763 | 0.1878 |
| Depth δ1 | 0.7588 | 0.7374 |
| Rotation AUC @ 5° | 0.1390 | 0.1298 |
| Rotation AUC @ 15° | 0.3860 | 0.4034 |
| Rotation AUC @ 30° | 0.5699 | 0.6301 |
| Translation AUC @ 5° | 0.0069 | 0.0056 |
| Translation AUC @ 15° | 0.0511 | 0.0412 |
| Translation AUC @ 30° | 0.1264 | 0.1232 |
| Joint pose AUC @ 5° | 0.0026 | 0.0010 |
| Joint pose AUC @ 15° | 0.0375 | 0.0291 |
| Joint pose AUC @ 30° | 0.1100 | 0.1109 |
Click an image for full size.
| Scene | Ground truth | Prediction |
|---|---|---|
| 1 | ![]() |
![]() |
| 2 | ![]() |
![]() |
| 3 | ![]() |
![]() |
| 4 | ![]() |
![]() |
Depth. Walls, floors and large furniture are captured, but thin objects like chair legs are lost, and flat walls look thick and noisy in the point clouds.
Rotation. Rough orientation is mostly right (AUC 0.57–0.63 at 30°), but precise alignment is not (about 0.13–0.14 at 5°).
Translation. This is the weakest result, with a mean direction error of 49.2° on validation and 46.8° on test, and it limits joint pose AUC. Predicted cameras bunch near the first camera instead of following the true path, most clearly in scene 4. A likely cause is that small translations are a low-loss choice when the model is uncertain.