First camera center: (0.0000, 0.0000, 0.0000)
Valid-point centroid distance: 1.000
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011280165199423209,
"max_ray_reprojection_error_px": 0.00010225349979009479,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization: All camera poses are expressed in the first camera's frame by multiplying them with the inverse of camera 0's pose, so camera 0 becomes the identity. Then translations, depth and points are divided by the distance from the origin to the centroid of all valid points, so every sample has the same scale (centroid distance 1).
World-space ray directions: For each pixel (u, v), K-1[u, v, 1] gives the ray direction in camera coordinates. Multiplying by the camera-to-world rotation R gives the direction in the canonical world frame, which is then normalized to unit length. The ray origin is the camera center.
Plücker moment: The moment is m = o × d, where o is the ray origin and d is the unit direction. It is the same for any point on the ray, so (d, m) describes the ray itself, not a particular point on it. Together, the 6 channels tell the model where each pixel's ray is in space.
Final overfit PSNR: 30.96 dB
Training settings: default configs/lvsm.json. 2 source views and 6 target views per sample, patch size 8, width 384, 8 layers, batch size 4, 100 epochs (5,600 steps), AdamW with learning rate 2e-4, weight decay 0.05, 200 warmup steps then cosine decay, bfloat16 autocast. Training took about 50 minutes on one A100.
Training curves:
Validation PSNR (dB) / SSIM:
| Source views | correct | shuffled | none |
|---|---|---|---|
| 1 | 15.19 / 0.558 | 11.80 / 0.488 | 12.47 / 0.529 |
| 2 | 16.14 / 0.560 | 11.49 / 0.472 | 12.44 / 0.528 |
| 4 | 16.90 / 0.575 | 11.59 / 0.473 | 12.44 / 0.538 |
Training visualization (validation images at epoch 95; columns: source views, ground truth, prediction):
Evaluation visualizations (columns in each image: source views, ground truth, prediction):
| correct | shuffled | none | |
|---|---|---|---|
| S=1 | ![]() | ![]() | ![]() |
| S=2 | ![]() | ![]() | ![]() |
| S=4 | ![]() | ![]() | ![]() |
Number of source views: PSNR increases from 15.19 dB with 1 source view to 16.14 dB with 2 and 16.90 dB with 4. More source views show more of the scene, so less of the target view has to be guessed.
Ray conditioning: With the correct target rays the model is 4-5 dB better than with shuffled or zero rays, so it uses the target camera to decide what to render. Shuffled rays are the worst because they point the model at the wrong viewpoint, while zero rays give no viewpoint, so the model outputs a blurry average. With wrong or missing rays, adding source views does not help.
The predictions are still blurry because 100 epochs is only 5,600 steps, so the model has mainly learned the coarse layout and colors of the scene.
Ground truth:
Predicted:
Training settings: default configs/vggt.json. Frozen DINOv2 ViT-B/14 encoder, 8 frame/global pairs, width 384, 6 heads, 8 views, sampling range 32, batch size 16, AdamW with learning rate 6e-4, weight decay 0.01, 10 warmup epochs then cosine decay, loss weights camera 5 and depth 1. Training was stopped at epoch 460 of the planned 1000 (about 4 hours on one A100); best.pt is from epoch 458 (lowest validation loss).
Training loss curves:
Metrics (8 views from the first 32 prepared frames, 224px, seed 825):
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel | 0.142 | 0.170 |
| Depth δ1 | 0.826 | 0.744 |
| Rot AUC@5 | 0.169 | 0.194 |
| Rot AUC@15 | 0.482 | 0.511 |
| Rot AUC@30 | 0.681 | 0.708 |
| Trans AUC@5 | 0.003 | 0.006 |
| Trans AUC@15 | 0.052 | 0.042 |
| Trans AUC@30 | 0.155 | 0.127 |
| Pose AUC@5 | 0.000 | 0.001 |
| Pose AUC@15 | 0.035 | 0.028 |
| Pose AUC@30 | 0.133 | 0.116 |
GT vs. predicted geometry on four validation scenes:
Scene 09c1414f1b_frame_000000. Ground truth (left), predicted (right):
Scene 21d970d8de_frame_000000. Ground truth (left), predicted (right):
Scene 280b83fcf3_frame_000000. Ground truth (left), predicted (right):
Scene 2b1dc6d6a5_frame_000000. Ground truth (left), predicted (right):
Errors on held-out scenes: Depth is the strongest part (δ1 0.83 on val, 0.74 on test): the room layout and large surfaces are correct, but the predicted depth is smooth and loses thin structures such as chair legs and table edges. Rotation is moderate (AUC@30 about 0.7, mean error about 10-13°). Translation direction is the weakest (mean error about 45°, AUC@30 0.15/0.13): the predicted cameras are bunched close to the first camera instead of spread out, so pose AUC is low. Test is slightly worse than val on depth. All losses were still decreasing at epoch 460, so training to 1000 epochs would likely improve the results.