Canonicalization. Transform cameras and points into the first sampled camera’s coordinate frame. Divide camera translations, depths and points by the distance from that camera to the valid-point centroid. Camera 0 becomes identity at (0, 0, 0), and the centroid distance becomes 1, as shown in the canonicalization screenshots above.
World-space ray directions. Backproject each pixel vector [u, v, 1] using the inverse intrinsics, rotate it with the camera-to-world rotation, and normalize it to unit length. This gives its direction in the canonical world frame. The camera center is the ray origin; translation affects the origin but not the direction.
Plücker moment. The moment is m = o × d, where o is the ray origin and d its unit direction. It encodes the line’s position relative to the coordinate origin and is unchanged when o moves along that line. The six channels are [dx, dy, dz, mx, my, mz].
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}Final PSNR: 36.70 dB.
Columns: source 1, source 2, ground truth, prediction.
Trained for 100 epochs on 224 scenes: 8 views sampled from 32-frame windows, 224 × 224 images, 2 source views, batch size 4, seed 825. The model uses 8 transformer blocks, width 384 and 8 × 8 patches. AdamW: learning rate 0.0002, weight decay 0.05, 200 warmup steps followed by cosine decay; BF16 forward passes with float32 loss. The checkpoint from epoch 94 was selected by validation PSNR.
Validation comparison saved during epoch 96 (val_epoch095.png); columns: source 1, source 2, ground truth, prediction.
Evaluation uses all 28 validation scenes, with 8 views from the first 32 prepared frames and seed 825.
| Source views | Target rays | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | Correct | 15.22 | 0.5568 |
| 1 | Shuffled | 11.80 | 0.4855 |
| 1 | None | 12.44 | 0.5290 |
| 2 | Correct | 16.19 | 0.5589 |
| 2 | Shuffled | 11.45 | 0.4705 |
| 2 | None | 12.42 | 0.5266 |
| 4 | Correct | 16.82 | 0.5739 |
| 4 | Shuffled | 11.58 | 0.4737 |
| 4 | None | 12.47 | 0.5372 |
With correct rays, increasing sources from 1 to 4 changed PSNR by +1.60 dB and SSIM by +0.0171; the highest PSNR used 4 sources. The target-image sets also change with the source count. At 2 sources, shuffling target rays changed PSNR/SSIM by -4.73 dB/-0.0884; removing them changed these by -3.77 dB/-0.0323. The saved validation comparison captures broad colors and layout, but predictions remain blurred and lose table/chair edges.
Columns: source view(s), ground truth, prediction.
1 source view, correct target rays.
1 source view, shuffled target rays.
1 source view, all-zero target rays.
2 source views, correct target rays.
2 source views, shuffled target rays.
2 source views, all-zero target rays.
4 source views, correct target rays.
4 source views, shuffled target rays.
4 source views, all-zero target rays.
MSE and MSE + 0.1 L1 were compared by fine-tuning the same best baseline weights for five additional epochs. Both branches used fresh AdamW optimizers at learning rate 0.00002, batch size 4, two source views, BF16 and seed 825, with identical ordered training scenes and frames in every epoch. The final checkpoints were evaluated on the same 28 validation scenes.
| Objective | PSNR (dB) | SSIM |
|---|---|---|
| Shared starting checkpoint | 16.1854 | 0.558877 |
| MSE continuation | 16.1591 | 0.558723 |
| MSE + 0.1 L1 | 16.1772 | 0.558515 |
Adding L1 changes PSNR by +0.0181 dB and SSIM by −0.000208 relative to the MSE continuation. Both remain below the starting checkpoint on both metrics. The matched examples remain blurred, with no clear recovery of fine furniture edges. This short comparison shows no clear overall gain; the small loss weight and brief fine-tuning may limit the change. One seed and five epochs do not establish performance after training from scratch.
Columns: source 1, source 2, ground truth, prediction. MSE continuation:
MSE + 0.1 L1 after the same five epochs:
Trained for 1,000 epochs (14,000 updates) on 224 scenes using 8 views from 32-frame windows, 224 × 224 images, batch size 16 and seed 825. Frozen DINOv2 ViT-B/14; 8 frame/global attention pairs, width 384, 6 heads; 4 camera-head blocks. AdamW: learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine decay to 5% of the initial rate. Total loss = 5 × camera loss + depth loss. BF16 forward passes with float32 losses; validation every 5 epochs. The best validation-loss checkpoint is epoch 850.
Full validation and test evaluation: 8 views from the first 32 prepared frames, 224px, seed 825, batch size 4. Each split contains 28 scenes and 784 valid camera pairs. AUC values range from 0 to 1.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.136629 | 0.163383 |
| Depth δ1 ↑ | 0.855322 | 0.799435 |
| Rotation AUC 5° ↑ | 0.197959 | 0.203316 |
| Rotation AUC 15° ↑ | 0.497874 | 0.520918 |
| Rotation AUC 30° ↑ | 0.688988 | 0.708121 |
| Translation AUC 5° ↑ | 0.008163 | 0.006122 |
| Translation AUC 15° ↑ | 0.067432 | 0.057738 |
| Translation AUC 30° ↑ | 0.170068 | 0.169430 |
| Joint pose AUC 5° ↑ | 0.001786 | 0.002551 |
| Joint pose AUC 15° ↑ | 0.050680 | 0.041752 |
| Joint pose AUC 30° ↑ | 0.149915 | 0.149575 |
The four held-out scenes retain broad layout, but predicted surfaces are thicker and warped. The tall structure in 280b83fcf3 is compressed and curved; the long pale wall in 21d970d8de bows; camera spacing and surface alignment differ in 2b1dc6d6a5. These views combine camera and depth errors, so they do not isolate either cause. Translation-direction AUC is much lower than rotation AUC, limiting joint pose accuracy. Test depth is worse than validation (AbsRel 0.1634 versus 0.1366).
The epoch-850 checkpoint, trained with 8 views, was evaluated with 2, 4, 8 and 16 nested input views on all 28 validation scenes. Each setting used the first 32 prepared frames, 224 × 224 images, seed 825 and batch size 1. Metrics are reported for all input views and for the same first two views to distinguish added context from changes in the evaluated frames.
| Views | All AbsRel ↓ | All δ1 ↑ | Common-two AbsRel ↓ | Common-two δ1 ↑ | Common-pair pose AUC 15° ↑ |
|---|---|---|---|---|---|
| 2 | 0.1199 | 0.8552 | 0.1199 | 0.8552 | 0.0119 |
| 4 | 0.1270 | 0.8622 | 0.1258 | 0.8591 | 0.0619 |
| 8 | 0.1367 | 0.8554 | 0.1325 | 0.8582 | 0.0762 |
| 16 | 0.1674 | 0.8005 | 0.1625 | 0.8142 | 0.0667 |
| Views | Median forward (ms) | Peak allocated (GiB) | Peak reserved (GiB) |
|---|---|---|---|
| 2 | 38.87 | 0.690 | 0.703 |
| 4 | 34.74 | 0.708 | 0.744 |
| 8 | 35.42 | 0.747 | 0.783 |
| 16 | 50.04 | 0.822 | 0.848 |
Measured on the L40S with BF16, three warmup forwards per setting and one synchronized forward per scene. Timing excludes data transfer, metrics and file I/O. Allocated memory includes model and inputs; reserved memory includes allocator caching. These are inference measurements.
Common-pair pose AUC at 15° improves from 0.0119 with 2 views to 0.0762 with 8, then falls to 0.0667 with 16. Common-view depth AbsRel worsens as views increase; δ1 peaks at 4 views. Sixteen views also use the most memory and take the longest. Additional context may help relative pose, while the change from the 8-view training setting may hurt depth. The loader recomputes scene normalization for each view count, so even the common-view comparison retains a scale confound. All-view metrics additionally use different frame/pair sets. One checkpoint, one seed and one timed pass per scene do not establish a general speed ordering for 2–8 views.
Matched frames 0 and 290 from scene 09c1414f1b, shown in meters with a shared 0–4.70 m range. Broad depth structure is retained, but chair legs, table edges and bookshelf details remain smoothed; added views show no clear detail recovery. White pixels are GT-invalid and masked in both depth panels.
2 input views:
16 input views: