junchif
Canonicalization transforms cameras and depth-unprojected points into the first sampled camera frame, then divides camera translations, depth and points by the distance from that camera to the valid-point centroid. The centroid is not subtracted. Camera 0 is identity and the normalized centroid distance is 1. World-space ray directions are normalize(R K⁻¹[u,v,1]), with camera-center origins. The Plücker moment m = o × d describes the ray line’s offset from the origin; channels are [d,m].





{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_005750",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9515330791473389,
"max_reprojection_error_px": 0.00012537825386971235,
"max_ray_reprojection_error_px": 0.00011620747682172805,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_003000",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9915946125984192,
"max_reprojection_error_px": 0.00010850114631466568,
"max_ray_reprojection_error_px": 9.650505671743304e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_007760",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9745520949363708,
"max_reprojection_error_px": 9.770392352947965e-05,
"max_ray_reprojection_error_px": 7.912623550510034e-05,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "4c5c60fa76_frame_010770",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9885328412055969,
"max_reprojection_error_px": 8.897326188161969e-05,
"max_ray_reprojection_error_px": 8.217109279939905e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
1,
8,
3,
224,
224
],
"depth": [
1,
8,
1,
224,
224
],
"depth_valid": [
1,
8,
1,
224,
224
],
"K": [
1,
8,
3,
3
],
"frame_numbers": [
1,
8
],
"T_world_from_camera": [
1,
8,
4,
4
],
"original_world_from_reference": [
1,
4,
4
],
"scene_scale": [
1
],
"points_world": [
1,
8,
3,
224,
224
],
"ray_origins": [
1,
8,
3,
224,
224
],
"ray_directions": [
1,
8,
3,
224,
224
],
"plucker": [
1,
8,
6,
224,
224
]
}
}One fixed batch, 2000 optimization steps. Final overfit PSNR: 36.59 dB.

| Training setting | Value |
|---|---|
| Input | 224×224; 8 views; sampling range 32; 2 sources and 6 targets |
| Model | Patch 8; width 384; 8 transformer blocks; attention head width 64 |
| Optimizer | AdamW; learning rate 0.0002; weight decay 0.05; betas (0.9,0.95) |
| Schedule | 100 epochs; batch size 2; warmup 200 steps; cosine decay; gradient clip 1 |
| Precision and seed | BF16 forward, FP32 loss; seed 825 |


| Source views | Target rays | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | correct | 15.07 | 0.5576 |
| 1 | shuffled | 11.83 | 0.4915 |
| 1 | none | 12.42 | 0.5379 |
| 2 | correct | 16.03 | 0.5591 |
| 2 | shuffled | 11.53 | 0.4764 |
| 2 | none | 12.41 | 0.5356 |
| 4 | correct | 16.66 | 0.5733 |
| 4 | shuffled | 11.65 | 0.4785 |
| 4 | none | 12.45 | 0.5467 |









With correct rays, PSNR increases from 15.07 to 16.03 to 16.66 dB for 1, 2 and 4 sources. The prescribed source/target partition also changes the target set, so this is not a fixed-target causal comparison. At two sources, shuffled and zero target rays reduce PSNR by 4.50 and 3.62 dB, respectively, supporting the importance of target-camera conditioning. Predictions retain broad color and light regions but blur chair legs, shelf contents and window edges.
Question: does adding L1 improve image quality? Compare MSE with MSE + 0.1 L1, using the same architecture, initial seed, sampling settings, optimizer and 100-epoch budget.
| Sources (correct rays) | MSE PSNR | MSE+L1 PSNR | MSE SSIM | MSE+L1 SSIM |
|---|---|---|---|---|
| 1 | 15.07 | 15.12 | 0.5576 | 0.5585 |
| 2 | 16.03 | 15.99 | 0.5591 | 0.5583 |
| 4 | 16.66 | 16.59 | 0.5733 | 0.5725 |


At two sources, L1 changes PSNR by -0.036 dB and SSIM by -0.0008. There is no consistent metric improvement across source counts. Both example predictions remain strongly blurred, without a convincing sharpness gain. L1 changes the pixel-error penalty but does not resolve this model’s missing fine detail. This is a single-seed experiment; the L1 run resumed after an interruption, preserving model/optimizer state but reseeding subsequent sample order, so the comparison is not perfectly matched in clip order.
One fixed training clip; 40 epochs × 25 steps, batch size 1 and learning rate 0.001.


| Training setting | Value |
|---|---|
| Input | 224×224; 8 views; sampling range 32 |
| Model | Frozen DINOv2 ViT-B/14; width 384; 8 frame/global pairs; 6 heads; camera and depth heads |
| Optimizer | AdamW; base learning rate 0.0006; weight decay 0.01; gradient clip 1 |
| Budget | 200 total epochs; batch size 4; seed 825; BF16 forward |
| Learning-rate schedule | 10-epoch warmup; original 1000-epoch plan through epoch 108; resumed with AdamW state and continuous cosine decay from 0.0005863292 to 0.00003 over epochs 109–200 |
| Loss weights | Camera 5; depth 1 |
| Evaluation | Full validation/test splits; 8 views from first 32 prepared frames; 224px; seed 825 |

| Split | Depth AbsRel ↓ | Depth δ1 ↑ | Threshold | Rotation AUC ↑ | Translation AUC ↑ | Joint AUC ↑ |
|---|---|---|---|---|---|---|
| Validation | 0.1647 | 0.7827 | 5° | 0.1449 | 0.0077 | 0.0003 |
| Validation | 0.1647 | 0.7827 | 15° | 0.3898 | 0.0494 | 0.0281 |
| Validation | 0.1647 | 0.7827 | 30° | 0.5773 | 0.1268 | 0.0999 |
| Test | 0.1833 | 0.7514 | 5° | 0.1684 | 0.0059 | 0.0026 |
| Test | 0.1833 | 0.7514 | 15° | 0.4313 | 0.0394 | 0.0274 |
| Test | 0.1833 | 0.7514 | 30° | 0.6372 | 0.1169 | 0.0990 |




Coarse room structure is recovered in 09c1414f1b, but predicted surfaces are thicker and less consistent. In 2b1dc6d6a5, points spread over a larger region than GT and camera-center arrangement differs. Depth AbsRel is 0.1647 on validation and 0.1833 on test, while mean translation-direction errors remain about 49° and 47°. Low joint-pose AUC indicates unreliable camera recovery despite plausible coarse depth. The 200-epoch resumed budget and the gap between fixed-clip overfitting and held-out scenes limit this result.
Question: how does input-view count affect accuracy, runtime and memory? Evaluate the same selected checkpoint with 2, 4 and 8 views, using the first 32 prepared validation frames, seed 825, 224px and batch size 1. No additional training is used. Inference timing includes input transfer, excludes an untimed warmup and synchronizes CUDA; memory is PyTorch peak allocated memory during evaluation.
| Views | Depth AbsRel ↓ | Depth δ1 ↑ | Joint AUC@30° ↑ | Inference ms/clip ↓ | Peak allocated GiB ↓ |
|---|---|---|---|---|---|
| 2 | 0.1302 | 0.8259 | 0.1024 | 42.15 | 0.691 |
| 4 | 0.1476 | 0.8065 | 0.1169 | 41.91 | 0.706 |
| 8 | 0.1647 | 0.7827 | 0.0999 | 56.87 | 0.755 |

Eight views use more memory and take longer than two or four views. The small two-versus-four timing difference is not persuasive evidence of a speed improvement in a single measurement. Fewer views have lower reported depth error here, but view selection, scene normalization and scored camera pairs also change (28, 168 and 784 pairs), so this does not establish that fewer views intrinsically improve reconstruction. Joint-pose accuracy remains weak for all counts. A fixed-frame/fixed-pair experiment and repeated timings would better isolate context size and runtime variability.