I backprojected each pixel with the inverse intrinsics, multiplied by z-depth, then applied the camera-to-world rotation and translation. I did not normalize the backprojected vector because depth is measured along the camera's z-axis.
Scene 036bce3393 with 8 views and frame 4470 as camera 0, in the original world frame in meters

I put all cameras and points in camera 0's frame, then divided translations, depth and points by the distance to the valid-point centroid. This makes the centroid distance 1 while keeping camera 0 at the origin. Invalid points stay zero.
The same scene and frames after canonicalization
Camera 0 center: (0, 0, 0)
Scale divisor: 1.867 m
Centroid distance: 1.000

I used the inverse intrinsics to get each pixel's camera direction, rotated it into the canonical world frame and normalized it. The ray starts at the camera center and points through the pixel. Camera translation moves the ray origin and camera rotation changes its direction.
I used [d, o x d], where d is the unit direction and o is the camera center. The moment o x d gives the line's position relative to the origin. It is perpendicular to the direction and stays the same if I move o along the ray. Camera 0 has zero moments because it is at the origin.
I checked four training samples. The largest point and ray reprojection errors were about 0.000111 pixels, below 0.01, and the centroid distances were all about 1.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Left to right has the source views, ground truth and prediction
I trained one fixed batch for 2000 steps using the default settings.
Final overfit PSNR: 30.878 dB
Final overfit SSIM: 0.8703
The table and chairs stay in roughly the right places as the target view changes, but their edges are softer than ground truth.

I trained for 100 epochs with 8 views, 2 sources, 224 by 224 images and batch size 4. I used 8 by 8 patches, width 384, 8 blocks and 6 heads. I used MSE and AdamW with learning rate 0.0002, 200 warmup steps and cosine decay. The other settings are from configs/lvsm.json, with a 32-frame sampling range, 32-frame evaluation prefix and seed 825.
Training and validation took about 47.5 minutes on the RTX 5060 Ti. I used the best checkpoint, from epoch 100.

Validation predictions from epoch 96 with two source columns:

The bright window and brown floor show up, but the table and chairs are blurred together. The square patches are also visible. The model gets the broad colors of the room but misses most of the detail.
I evaluated all 28 validation scenes with correct target rays, another target's rays (shuffled), or zero rays (none).
| Sources | Target rays | PSNR (dB) | SSIM | Target images | Visualization |
|---|---|---|---|---|---|
| 1 | correct | 15.238 | 0.5584 | 196 | image |
| 1 | shuffled | 11.753 | 0.4845 | 196 | image |
| 1 | none | 12.479 | 0.5241 | 196 | image |
| 2 | correct | 16.150 | 0.5602 | 168 | image |
| 2 | shuffled | 11.430 | 0.4698 | 168 | image |
| 2 | none | 12.458 | 0.5240 | 168 | image |
| 4 | correct | 16.889 | 0.5760 | 112 | image |
| 4 | shuffled | 11.561 | 0.4728 | 112 | image |
| 4 | none | 12.472 | 0.5345 | 112 | image |
PSNR and SSIM go up with more source views when the rays are correct. More sources show more of the room. However, the remaining views become targets, so changing the number of sources also changes which images get evaluated. This could account for part of the increase.
One source, correct rays:

Two sources, correct rays:

Four sources, correct rays:

Even with four sources, the furniture and texture are much smoother than ground truth.
With two sources, shuffling the rays drops PSNR by about 4.72 dB and zeroing them drops it by 3.69 dB. Giving the model the wrong viewpoint is worse here than giving it no target rays.
Two sources, shuffled rays:

Two sources, zero rays:

With zero rays, every row has the same prediction even though ground truth changes. The model gets the same source views and query tokens for every target.
Full ablation metrics and training log
I added 0.1 * L1 to MSE to see if it would make the predictions sharper. I trained for 100 epochs with the same model and settings as 2.2, including seed 825. Both best checkpoints were from epoch 100. I compared them on the same 168 target images, with two sources and correct rays.
| Training loss | Validation PSNR (dB) | Validation SSIM | Logged train + validation time |
|---|---|---|---|
| MSE | 16.150 | 0.5602 | 47.5 min |
| MSE + 0.1 L1 | 16.215 | 0.5605 | 47.6 min |

MSE with two source columns:

MSE + 0.1 L1 using the same views:

Adding L1 improved PSNR by only 0.065 dB and SSIM by 0.0003. The curves almost overlap, and the window and furniture look about as blurry in both images. Both losses compare pixels and adding this much L1 did not seem to help the missing detail. I only tried one weight and one seed, so the small score increase may not hold in another run.
Settings: configs/lvsm.json and configs/lvsm_l1.json
I trained on 8 fixed frames from scene 036bce3393 for 40 epochs with 25 updates per epoch and learning rate 0.001. I used the checkpoint from epoch 40.
Depth AbsRel: 0.619 -> 0.0379
Depth delta1: 0.208 -> 0.9533
Final mean rotation error: 0.287 degrees
Final mean translation-direction error: 0.575 degrees

The depth and total losses become negative because the confidence term includes -0.2 * log(confidence).
Ground truth:

Prediction:

The predicted cloud and cameras are close to GT. The depth map is smoother, but the room's shape is much closer than in the held-out scenes below.
I trained on all 224 training scenes for 1000 epochs using the default model in configs/vggt.json. I kept DINOv2 ViT-B/14 frozen and used 8 frame/global attention pairs with width 384 and 6 heads. I used 8 views, sampling range 32, 224 by 224 images, batch size 16 and seed 825. I used AdamW with learning rate 0.0006, weight decay 0.01, 10 warmup epochs and cosine decay.
I started the transformer and heads from scratch. Training and validation took about 4.73 hours on the RTX 5060 Ti.

Camera and depth loss rise around epoch 710 and do not fully recover. The best validation loss was 0.8890 at epoch 671, so I used that checkpoint for evaluation.
I evaluated all 28 validation scenes and all 28 test scenes using 8 views from the first 32 prepared frames, 224 by 224 images and seed 825.
| Split | Depth AbsRel | Depth delta1 |
|---|---|---|
| Validation | 0.1660 | 0.7801 |
| Test | 0.1846 | 0.7555 |
| Split | Pose metric | AUC at 5 degrees | AUC at 15 degrees | AUC at 30 degrees |
|---|---|---|---|---|
| Validation | Rotation | 0.1834 | 0.5097 | 0.6933 |
| Validation | Translation direction | 0.0084 | 0.0755 | 0.1894 |
| Validation | Joint pose | 0.0026 | 0.0554 | 0.1638 |
| Test | Rotation | 0.1747 | 0.4843 | 0.6918 |
| Test | Translation direction | 0.0041 | 0.0380 | 0.1299 |
| Test | Joint pose | 0.0008 | 0.0263 | 0.1202 |
Translation direction scores much worse than rotation on both splits. Test depth is also worse than validation. Errors in the cameras move the point clouds from different views apart, even if the depth maps look reasonable.
Ground truth first and prediction second for each scene, with the same viewing angle and axes
Scene 1:


The dining table is in roughly the right place, but its edges and the chairs are blurred in the predicted depth.
Scene 2:


The floor and wall show up, but the thin furniture parts blend into the surrounding surfaces.
Scene 3:


The hallway walls should be straight, but they curve and blend together in the prediction.
Scene 4:


The office cloud is more spread out than GT, and the chair is harder to make out. The model could memorize the training clip, but the held-out rooms have much larger errors.
Validation metrics, test metrics and training history
I varied the number of input views to be 2, 4 or 8 using the same model from epoch 671. I used all 28 validation scenes with the same frame prefix, resolution and seed as 3.2. The 2-view input uses the first 2 views of the 4-view input, and the 4-view input uses the first 4 views of the 8-view input. Frame 0 stays the reference.
For timing, I used one fixed scene at batch size 1 and averaged five GPU forward passes after two warmups. This excludes loading and metrics. Memory is peak PyTorch allocation, including model weights.
| Views | Depth AbsRel | Depth delta1 | Joint AUC at 15 degrees | Inference (ms) | Peak allocation (GB) |
|---|---|---|---|---|---|
| 2 | 0.1311 | 0.8540 | 0.0548 | 12.27 | 0.872 |
| 4 | 0.1494 | 0.8249 | 0.0520 | 17.24 | 0.883 |
| 8 | 0.1660 | 0.7801 | 0.0554 | 26.93 | 0.910 |

Eight views take about twice as long as two views, from 12.3 to 26.9 ms. Memory only goes from 0.872 to 0.910 GB because the model weights stay the same.
Two views score better on depth, while joint-pose AUC barely changes. However, fewer views also means fewer images and pose pairs get evaluated, and the loader rescales geometry from the chosen views. This makes it hard to tell whether fewer views actually help the model.
The same dining room with different numbers of views, each normalized separately:
2 views:

4 views:

8 views:

More views fill in more of the room, but the points still do not line up into clean surfaces. I only changed the views during evaluation, so training with fewer views might give different results. The timing is also just for one scene.
Full view-count metrics and timing trials. Run scripts/benchmark_vggt_views.py with TORCH_HOME=data/torch and the checkpoint at outputs/vggt_train/best.pt to repeat the experiment.