
Unprojection 1 Image

Unprojection 2 Image

Canonicalization 1 Image

Canonicalization 2 Image
{
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
}
Carnonicalization makes each sample in the first image's camera frame and normalize by the scene scale. Camera 0 is at origin, and valid-point centroid 1 from origin. A ray direction is a unit vector that go through the camera center, from pixel to the corresponding point. The moment is o x d. So rays with the same direction but different origins are different.
psnr 30.96

val_epoch000 Image

training_curves Image
{
S=1,conditioning=correct: PSNR 15.20 SSIM 0.5589 loss 0.03514 (196 images)
S=1,conditioning=shuffled: PSNR 11.80 SSIM 0.4879 loss 0.07296 (196 images)
S=1,conditioning=none: PSNR 12.48 SSIM 0.5298 loss 0.05872 (196 images)
S=2,conditioning=correct: PSNR 16.13 SSIM 0.5606 loss 0.02940 (168 images)
S=2,conditioning=shuffled: PSNR 11.49 SSIM 0.4723 loss 0.07785 (168 images)
S=2,conditioning=none: PSNR 12.45 SSIM 0.5281 loss 0.05903 (168 images)
S=4,conditioning=correct: PSNR 16.87 SSIM 0.5761 loss 0.02469 (112 images)
S=4,conditioning=shuffled: PSNR 11.59 SSIM 0.4733 loss 0.07842 (112 images)
S=4,conditioning=none: PSNR 12.46 SSIM 0.5383 loss 0.05881 (112 images)
}

1 view conditioning correct Image

2 view conditioning correct Image

4 view conditioning correct Image
More source views improve the results when the target rays are correct, things are more likely at the right place. Shuffled rays are worse than no rays, because it generate another target view. The none one has no camera conditioning, so would tend to generate a generic, mean image, which is not good but favor the metrics.
Question: whether adding L1 loss can improve the results. Other experimental settings are the same.
{
S=1,conditioning=correct: PSNR 15.34 SSIM 0.5600 loss 0.03460 (196 images)
S=2,conditioning=correct: PSNR 16.29 SSIM 0.5636 loss 0.02892 (168 images)
S=4,conditioning=correct: PSNR 17.11 SSIM 0.5802 loss 0.02375 (112 images)
}

training_curves Image

1 view conditioning correct Image

2 view conditioning correct Image

4 view conditioning correct Image
The PSNR and SSIM are slightly improved, but from the visuals cannot see much difference. So L1 loss is not much helpful. May be adding other percptual loss will be better.

loss curves Image

GT Image

Pred Image

loss curves Image
Validation results:
{
"split": "val",
"checkpoint": "outputs/vggt_train/best.pt",
"max_batches": null,
"loss": 0.8353907210486275,
"loss_camera": 0.11057302036455699,
"loss_depth": 0.2825255947453635,
"num_samples": 28,
"num_valid_pixels": 10347720,
"num_pose_pairs": 784,
"num_degenerate_pose_pairs": 0,
"depth_abs_rel": 0.15094577110865448,
"depth_rmse_normalized": 0.20340209165434642,
"depth_delta1": 0.8020720506546369,
"rotation_error_deg": 12.121335830446727,
"translation_error_deg": 40.64689853176176,
"rotation_auc_3": 0.11181972789115646,
"rotation_auc_5": 0.20739795918367349,
"rotation_auc_15": 0.533248299319728,
"rotation_auc_30": 0.7108418367346939,
"translation_auc_3": 0.006802721088435374,
"translation_auc_5": 0.01607142857142857,
"translation_auc_15": 0.08239795918367347,
"translation_auc_30": 0.19477040816326532,
"pose_auc_3": 0.0017006802721088435,
"pose_auc_5": 0.005867346938775509,
"pose_auc_15": 0.05756802721088436,
"pose_auc_30": 0.16679421768707486
}
Test results:
{
"split": "test",
"checkpoint": "outputs/vggt_train/best.pt",
"max_batches": null,
"loss": 0.7958150506019592,
"loss_camera": 0.09793056015457426,
"loss_depth": 0.306162280695779,
"num_samples": 28,
"num_valid_pixels": 10517881,
"num_pose_pairs": 784,
"num_degenerate_pose_pairs": 0,
"depth_abs_rel": 0.16838197852559653,
"depth_rmse_normalized": 0.21677884981718784,
"depth_delta1": 0.759972659892235,
"rotation_error_deg": 9.912632167535051,
"translation_error_deg": 42.34491963970228,
"rotation_auc_3": 0.11819727891156463,
"rotation_auc_5": 0.20637755102040817,
"rotation_auc_15": 0.5267006802721089,
"rotation_auc_30": 0.7176445578231294,
"translation_auc_3": 0.004251700680272109,
"translation_auc_5": 0.010459183673469388,
"translation_auc_15": 0.06249999999999999,
"translation_auc_30": 0.16641156462585033,
"pose_auc_3": 0.0012755102040816326,
"pose_auc_5": 0.004081632653061224,
"pose_auc_15": 0.04600340136054422,
"pose_auc_30": 0.1483418367346939
}

GT1 Image

Pred1 Image

GT2 Image

Pred3 Image

GT3 Image

Pred3 Image

GT4 Image

Pred4 Image
I am following the default training settings. The precitions on held-out scenes can show rough structures, but it's precision is much worse.