Assignment 4

16-825 Learning for 3D Vision

Task 1.2

Unprojection 1 Image

Unprojection 1 Image

Unprojection 2 Image

Unprojection 2 Image

Task 1.3

Carnonization 1 Image

Canonicalization 1 Image

Carnonization 2 Image

Canonicalization 2 Image

Task 1.7

{
    {
      "split": "train",
      "dataset_samples": 224,
      "checked_samples": 4,
      "samples": [
        {
          "sequence_name": "e050c15a8d_frame_007080",
          "views": 8,
          "resolution": 224,
          "valid_depth_fraction": 0.9779127240180969,
          "max_reprojection_error_px": 0.00011009880108758807,
          "max_ray_reprojection_error_px": 0.00010764275066321716,
          "centroid_distance": 1.0
        },
        {
          "sequence_name": "5f99900f09_frame_002820",
          "views": 8,
          "resolution": 224,
          "valid_depth_fraction": 0.9903116226196289,
          "max_reprojection_error_px": 0.00012302029062993824,
          "max_ray_reprojection_error_px": 0.00011108566104667261,
          "centroid_distance": 0.9999999403953552
        },
        {
          "sequence_name": "47b37eb6f9_frame_002720",
          "views": 8,
          "resolution": 224,
          "valid_depth_fraction": 0.9609947800636292,
          "max_reprojection_error_px": 9.881925507215783e-05,
          "max_ray_reprojection_error_px": 8.897326188161969e-05,
          "centroid_distance": 1.0
        },
        {
          "sequence_name": "4c5c60fa76_frame_006690",
          "views": 8,
          "resolution": 224,
          "valid_depth_fraction": 0.7516716122627258,
          "max_reprojection_error_px": 0.0001089862416847609,
          "max_ray_reprojection_error_px": 0.00010257320536766201,
          "centroid_distance": 0.9999999403953552
        }
      ],
      "batch_shapes": {
        "rgb": [
          2,
          8,
          3,
          224,
          224
        ],
        "depth": [
          2,
          8,
          1,
          224,
          224
        ],
        "depth_valid": [
          2,
          8,
          1,
          224,
          224
        ],
        "K": [
          2,
          8,
          3,
          3
        ],
        "frame_numbers": [
          2,
          8
        ],
        "T_world_from_camera": [
          2,
          8,
          4,
          4
        ],
        "original_world_from_reference": [
          2,
          4,
          4
        ],
        "scene_scale": [
          2
        ],
        "points_world": [
          2,
          8,
          3,
          224,
          224
        ],
        "ray_origins": [
          2,
          8,
          3,
          224,
          224
        ],
        "ray_directions": [
          2,
          8,
          3,
          224,
          224
        ],
        "plucker": [
          2,
          8,
          6,
          224,
          224
        ]
      }
    }
}

Carnonicalization makes each sample in the first image's camera frame and normalize by the scene scale. Camera 0 is at origin, and valid-point centroid 1 from origin. A ray direction is a unit vector that go through the camera center, from pixel to the corresponding point. The moment is o x d. So rays with the same direction but different origins are different.

Task 2.1

psnr 30.96

val_epoch000 Image

val_epoch000 Image

Task 2.2

training_curves Image

training_curves Image

{
  S=1,conditioning=correct: PSNR 15.20  SSIM 0.5589  loss 0.03514  (196 images)
  S=1,conditioning=shuffled: PSNR 11.80  SSIM 0.4879  loss 0.07296  (196 images)
  S=1,conditioning=none: PSNR 12.48  SSIM 0.5298  loss 0.05872  (196 images)
  S=2,conditioning=correct: PSNR 16.13  SSIM 0.5606  loss 0.02940  (168 images)
  S=2,conditioning=shuffled: PSNR 11.49  SSIM 0.4723  loss 0.07785  (168 images)
  S=2,conditioning=none: PSNR 12.45  SSIM 0.5281  loss 0.05903  (168 images)
  S=4,conditioning=correct: PSNR 16.87  SSIM 0.5761  loss 0.02469  (112 images)
  S=4,conditioning=shuffled: PSNR 11.59  SSIM 0.4733  loss 0.07842  (112 images)
  S=4,conditioning=none: PSNR 12.46  SSIM 0.5383  loss 0.05881  (112 images)
}

S1_correct Image

1 view conditioning correct Image

S2_correct Image

2 view conditioning correct Image

S4_correct Image

4 view conditioning correct Image

More source views improve the results when the target rays are correct, things are more likely at the right place. Shuffled rays are worse than no rays, because it generate another target view. The none one has no camera conditioning, so would tend to generate a generic, mean image, which is not good but favor the metrics.

Task 2.3

Question: whether adding L1 loss can improve the results. Other experimental settings are the same.

{
  S=1,conditioning=correct: PSNR 15.34  SSIM 0.5600  loss 0.03460  (196 images)
  S=2,conditioning=correct: PSNR 16.29  SSIM 0.5636  loss 0.02892  (168 images)
  S=4,conditioning=correct: PSNR 17.11  SSIM 0.5802  loss 0.02375  (112 images)
}

training_curves Image

training_curves Image

S1_correct Image

1 view conditioning correct Image

S2_correct Image

2 view conditioning correct Image

S4_correct Image

4 view conditioning correct Image

The PSNR and SSIM are slightly improved, but from the visuals cannot see much difference. So L1 loss is not much helpful. May be adding other percptual loss will be better.

Task 3.1

loss_curves Image

loss curves Image

gt Image

GT Image

pred Image

Pred Image

Task 3.2

loss_curves Image

loss curves Image

Validation results:

{
  "split": "val",
  "checkpoint": "outputs/vggt_train/best.pt",
  "max_batches": null,
  "loss": 0.8353907210486275,
  "loss_camera": 0.11057302036455699,
  "loss_depth": 0.2825255947453635,
  "num_samples": 28,
  "num_valid_pixels": 10347720,
  "num_pose_pairs": 784,
  "num_degenerate_pose_pairs": 0,
  "depth_abs_rel": 0.15094577110865448,
  "depth_rmse_normalized": 0.20340209165434642,
  "depth_delta1": 0.8020720506546369,
  "rotation_error_deg": 12.121335830446727,
  "translation_error_deg": 40.64689853176176,
  "rotation_auc_3": 0.11181972789115646,
  "rotation_auc_5": 0.20739795918367349,
  "rotation_auc_15": 0.533248299319728,
  "rotation_auc_30": 0.7108418367346939,
  "translation_auc_3": 0.006802721088435374,
  "translation_auc_5": 0.01607142857142857,
  "translation_auc_15": 0.08239795918367347,
  "translation_auc_30": 0.19477040816326532,
  "pose_auc_3": 0.0017006802721088435,
  "pose_auc_5": 0.005867346938775509,
  "pose_auc_15": 0.05756802721088436,
  "pose_auc_30": 0.16679421768707486
}

Test results:

{
  "split": "test",
  "checkpoint": "outputs/vggt_train/best.pt",
  "max_batches": null,
  "loss": 0.7958150506019592,
  "loss_camera": 0.09793056015457426,
  "loss_depth": 0.306162280695779,
  "num_samples": 28,
  "num_valid_pixels": 10517881,
  "num_pose_pairs": 784,
  "num_degenerate_pose_pairs": 0,
  "depth_abs_rel": 0.16838197852559653,
  "depth_rmse_normalized": 0.21677884981718784,
  "depth_delta1": 0.759972659892235,
  "rotation_error_deg": 9.912632167535051,
  "translation_error_deg": 42.34491963970228,
  "rotation_auc_3": 0.11819727891156463,
  "rotation_auc_5": 0.20637755102040817,
  "rotation_auc_15": 0.5267006802721089,
  "rotation_auc_30": 0.7176445578231294,
  "translation_auc_3": 0.004251700680272109,
  "translation_auc_5": 0.010459183673469388,
  "translation_auc_15": 0.06249999999999999,
  "translation_auc_30": 0.16641156462585033,
  "pose_auc_3": 0.0012755102040816326,
  "pose_auc_5": 0.004081632653061224,
  "pose_auc_15": 0.04600340136054422,
  "pose_auc_30": 0.1483418367346939
}

gt1 Image

GT1 Image

pred1 Image

Pred1 Image

gt2 Image

GT2 Image

pred2 Image

Pred3 Image

gt3 Image

GT3 Image

pred3 Image

Pred3 Image

gt4 Image

GT4 Image

pred4 Image

Pred4 Image

I am following the default training settings. The precitions on held-out scenes can show rough structures, but it's precision is much worse.