1. Multi-View Data Preprocessing (20 points)¶

unprojection:

unprojection

canonicalization:

canonicalization

check_data.py output:

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 9.448717901250347e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.591897105565295e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001089862416847609,
      "max_ray_reprojection_error_px": 0.00010141761595150456,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}
  • Canonicalization: Transform camera poses and pointclouds to an unified coodinates frame for all scenes.
  • World-space ray directions: The unit vector of camera rays cast from camera center through each pixel in the canonical coordinate frame.
  • The Plücker moment: The vector component $\mathbf{m} = \mathbf{o} \times \mathbf{d}$ of a 3D ray, it can represent 3D lines without the raw origin.

2. Novel-View Synthesis with Mini-LVSM (40 points)¶

2.1. Overfitting (10 points)¶

PSNR: 30.94

val_epoch000

2.2. Training and Evaluation (30 points)¶

the training curves:¶

traning_curves

num-source-views conditioning PSNR (dB) SSIM
1 correct 15.20 0.558
1 shuffled 11.80 0.488
1 none 12.47 0.529
2 correct 16.14 0.560
2 shuffled 11.50 0.473
2 none 12.45 0.527
4 correct 16.88 0.575
4 shuffled 11.60 0.473
4 none 12.45 0.537

the training visualizations:¶

val_epoch000 val_epoch095

evaluation visualizations:¶

S1_correct:

S1_correct

S1_none:

S1_none

S1_shuffled:

S1_shuffled

S2_correct:

S2_correct

S2_none:

S2_none

S2_shuffled:

S2_shuffled

S4_correct:

S4_correct

S4_none:

S4_none

S4_shuffled:

S4_shuffled

  • More number of source views increase the result of PSNR.
  • The correct ray-conditioning ablation lead to the better performance than other two condition(shuffled and none).

2.3. Bonus: Analysis and Exploration (10 points)¶

Model capacity: How do the patch size, transformer width or depth affect quality, runtime and memory?

I choose to test with patch_size. I tried the value of 8, 16, 32. The results showed that both RSNR and SSIM decrease when the patch_size increase, but the runtime and memory drop significantly.

Smaller patches give each token detailed information, so the reconstruction is better. And the limitation is patch_size can't be too small, need to ensure the hardware computational resource before training.

evaluation visualization:

patch_size=8:

S2_correct

patch_size=16:

S2_correct

patch_size=32:

S2_correct

3. Multi-View Geometry with Mini-VGGT (40 points)¶

3.1. Overfitting (10 points)¶

the overfit loss curve image:

loss_curve_overfit

GT:¶

overfit_gt

predicted geometry:¶

overfit_pred

3.2. Training and Evaluation (30 points)¶

loss_curve

Metric Val Test Better
Depth AbsRel 0.150 0.157 lower
Depth δ1 0.809 0.791 higher
Rotation AUC @5° / @15° / @30° 0.190 / 0.535 / 0.710 0.223 / 0.550 / 0.741 higher
Translation AUC @5° / @15° / @30° 0.008 / 0.070 / 0.177 0.017 / 0.101 / 0.229 higher
Joint pose AUC @5° / @15° / @30° 0.003 / 0.053 / 0.154 0.003 / 0.077 / 0.211 higher

comparing GT and predicted geometry for four different scenes:

scene 09c1414f1b:¶

  • GT:

scene09_gt

  • prediction:

scene09_pred

scene 21d970d8de:¶

  • GT:

scene21_gt

  • prediction:

scene21_pred

scene 280b83fcf3:¶

  • GT:

scene28_gt

  • prediction:

scene28_pred

scene 2b1dc6d6a5:¶

  • GT:

scene2b_gt

  • prediction:

scene2b_pred

The result shows that accuracy improvements in rotation estimation contribute more significantly to the overall pose performance than translation estimation. The reason might be the projection is sensitive to orientation, because the far distance pixel will shift multiple times than the near one.

3.3. Bonus: Analysis and Exploration (10 points)¶

Number of views: How does the number of input views affect accuracy, runtime and memory? Explore changes during training and/or evaluation.

I tested with different number of views, and I haved tried the value=4, 8, 16.

Val:¶

num_views Depth AbsRel Depth δ1 Rot AUC @5/15/30 Trans AUC @5/15/30 Joint AUC@30
4 0.129 0.849 0.213 / 0.562 / 0.715 0.004 / 0.061 / 0.165 0.140
8 0.150 0.809 0.190 / 0.535 / 0.710 0.008 / 0.070 / 0.177 0.154
16 0.179 0.769 0.183 / 0.487 / 0.683 0.009 / 0.063 / 0.164 0.137

Test:¶

num_views Depth AbsRel Depth δ1 Rot AUC @5/15/30 Trans AUC @5/15/30 Joint AUC@30
4 0.145 0.819 0.261 / 0.573 / 0.759 0.013 / 0.106 / 0.216 0.191
8 0.157 0.791 0.223 / 0.550 / 0.741 0.017 / 0.101 / 0.229 0.211
16 0.169 0.764 0.228 / 0.557 / 0.741 0.013 / 0.097 / 0.232 0.215

The result shows that depth prediction get worse as num_views grows. The reason might be the model was trained in 8 views. When the num_views increase to 16, there are some views that the model never saw, so the depth prediction drift. Also, as the num_views increases, Rotation angle accuracy drops. The cause might be the severe self-occlusion and viewpoint deformation caused by the larger angles between views.

Evaluating with num_views only indicate the model's generalization to unseen view, not the optimal performance of this model. If we want to know the optimal performance of the num_views actually affect, the training should be done by different num_views then do the evaluation.