| unprojection | canonicalization |
|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
check_data.py outputpython scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010844774078577757,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.281559323426336e-05,
"max_ray_reprojection_error_px": 8.446346328128129e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010504279634915292,
"max_ray_reprojection_error_px": 9.423537994734943e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization. Canonicalization is to put every views into the first view, so camera 0 becomes I. Other frames are projected into this.
World-space ray directions. For pixel
(u, v) the camera-space direction is
K^-1 [u, v, 1]^T. Rotating it by the camera-to-world
rotation R and normalizing gives the world-space unit
direction d.
Plücker moment. The moment is
m = o * d. Any other point on the same ray gives the same moment, so (d, m)
describes the ray itself rather than a particular point on it.
Final overfit PSNR: 30.83 dB (SSIM 0.870).

Training settings:

| source views S | correct | shuffled | none |
|---|---|---|---|
| 1 | 15.13 / 0.557 | 11.82 / 0.489 | 12.44 / 0.531 |
| 2 | 16.05 / 0.558 | 11.49 / 0.474 | 12.43 / 0.529 |
| 4 | 16.68 / 0.573 | 11.63 / 0.478 | 12.48 / 0.539 |
Columns: source views, ground truth, prediction.

Columns: source views, ground truth, prediction.
| S | correct | shuffled | none |
|---|---|---|---|
| 1 | ![]() |
![]() |
![]() |
| 2 | ![]() |
![]() |
![]() |
| 4 | ![]() |
![]() |
![]() |
Number of source views. With the correct rays, PSNR goes from 15.13 (S = 1) to 16.05 (S = 2) to 16.68 dB (S = 4), even though the model was trained with S = 2 only. More source views cover more of the scene that the target camera sees, so less of the target has to be guessed.
Ray conditioning. Without target rays
the model prediction starts to get blurry, roughly
average-colored image for every target, about 12.4 dB. With another
view's rays it renders a scene layout that
belongs to a different viewpoint, so windows and furniture show up in
the wrong places. This is even worse than none (about
11.5-11.8 dB), because putting structure in the wrong place costs more
MSE than predicting a blurry average.
| loss curve (total, camera, depth) |
|---|
![]() |
| input view 0 | GT geometry | predicted geometry |
|---|---|---|
![]() |
![]() |
![]() |
My training settings:

8 views from the first 32 prepared frames, 224px, seed 825.
| split | AbsRel ↓ | δ1 ↑ | rotation AUC @5/15/30 ↑ | translation AUC @5/15/30 ↑ | pose AUC @5/15/30 ↑ |
|---|---|---|---|---|---|
| val | 0.163 | 0.793 | 0.187 / 0.502 / 0.690 | 0.007 / 0.050 / 0.148 | 0.004 / 0.041 / 0.136 |
| test | 0.172 | 0.792 | 0.190 / 0.498 / 0.702 | 0.002 / 0.051 / 0.157 | 0.000 / 0.036 / 0.140 |
| input view 0 | GT geometry | predicted geometry |
|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
I visualized a few samples as if they were in the viewpoint of the camera. Comparing Ground Truth and the Prediction, I can see the points prediction seems to be blend toghther (unlike GT, it has a clear contour for example chairs).