Canonicalization: apply the inverse of camera 0's pose, then divide translations, depth and points by s = ‖mean(p_valid)‖. Camera 0 becomes identity and the valid-point centroid is one unit from the origin.
World-space ray directions: normalize R K⁻¹[u, v, 1]ᵀ, where R is the camera-to-world rotation. The ray origin is the camera center.
Plücker coordinates: [d, m], where d is the unit ray direction and m = o × d is the moment, encoding the ray's offset from the origin.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 9.41436446737498e-05, "centroid_distance": 1.0},
{"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00011108566104667261, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
{"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.591897105565295e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
{"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.0001089862416847609, "max_ray_reprojection_error_px": 0.00010141761595150456, "centroid_distance": 0.9999999403953552}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Final overfit PSNR: 30.04 dB.
Training settings: 100 epochs; batch size 4; 2 source views; 8 total views at 224×224; sampling range 32; AdamW; learning rate 0.0002; weight decay 0.05; 200 warmup steps; cosine decay; seed 825.
More source views improve reconstruction by providing more observations. Shuffling or zeroing target rays reduces PSNR and SSIM, showing dependence on camera conditioning; shuffled rays are more harmful here.
S=1, correct rays: PSNR 15.19 dB; SSIM 0.5579.
S=1, shuffled rays: PSNR 11.80 dB; SSIM 0.4871.
S=1, zeroed rays: PSNR 12.46 dB; SSIM 0.5302.
S=2, correct rays: PSNR 16.14 dB; SSIM 0.5602.
S=2, shuffled rays: PSNR 11.50 dB; SSIM 0.4722.
S=2, zeroed rays: PSNR 12.44 dB; SSIM 0.5274.
S=4, correct rays: PSNR 16.88 dB; SSIM 0.5756.
S=4, shuffled rays: PSNR 11.59 dB; SSIM 0.4732.
S=4, zeroed rays: PSNR 12.45 dB; SSIM 0.5372.
Ground truth
Prediction
Training settings: 1,000 epochs; batch size 16; 8 views at 224×224; sampling range 32; AdamW; learning rate 0.0006; weight decay 0.01; 10 warmup epochs; cosine decay; camera/depth loss weights 5/1; seed 825.
Validation: depth AbsRel 0.1443; depth δ1 81.58%.
Validation rotation AUC — 5°: 0.1893; 15°: 0.5167; 30°: 0.6939.
Validation translation AUC — 5°: 0.0133; 15°: 0.0912; 30°: 0.2019.
Validation joint pose AUC — 5°: 0.0038; 15°: 0.0753; 30°: 0.1843.
Test: depth AbsRel 0.1557; depth δ1 79.14%.
Test rotation AUC — 5°: 0.2434; 15°: 0.5707; 30°: 0.7584.
Test translation AUC — 5°: 0.0168; 15°: 0.0720; 30°: 0.1770.
Test joint pose AUC — 5°: 0.0061; 15°: 0.0575; 30°: 0.1679.
On held out scenes, depth recovers broad room surfaces but loses fine structure. Camera errors, especially translation, misalign points across views and reduce joint-pose AUC on held-out scenes.
Ground truth
Prediction
Ground truth
Prediction
Ground truth
Prediction
Ground truth
Prediction