1. Multi-View Data Preprocessing (20 points)¶
unprojection:

canonicalization:

check_data.py output:
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.448717901250347e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
- Canonicalization: Transform camera poses and pointclouds to an unified coodinates frame for all scenes.
- World-space ray directions: The unit vector of camera rays cast from camera center through each pixel in the canonical coordinate frame.
- The Plücker moment: The vector component $\mathbf{m} = \mathbf{o} \times \mathbf{d}$ of a 3D ray, it can represent 3D lines without the raw origin.
2. Novel-View Synthesis with Mini-LVSM (40 points)¶
2.1. Overfitting (10 points)¶
PSNR: 30.94

2.2. Training and Evaluation (30 points)¶
the training curves:¶

| num-source-views | conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.20 | 0.558 |
| 1 | shuffled | 11.80 | 0.488 |
| 1 | none | 12.47 | 0.529 |
| 2 | correct | 16.14 | 0.560 |
| 2 | shuffled | 11.50 | 0.473 |
| 2 | none | 12.45 | 0.527 |
| 4 | correct | 16.88 | 0.575 |
| 4 | shuffled | 11.60 | 0.473 |
| 4 | none | 12.45 | 0.537 |
the training visualizations:¶

evaluation visualizations:¶
S1_correct:

S1_none:

S1_shuffled:

S2_correct:

S2_none:

S2_shuffled:

S4_correct:

S4_none:

S4_shuffled:

- More number of source views increase the result of PSNR.
- The correct ray-conditioning ablation lead to the better performance than other two condition(shuffled and none).
2.3. Bonus: Analysis and Exploration (10 points)¶
Model capacity: How do the patch size, transformer width or depth affect quality, runtime and memory?
I choose to test with patch_size. I tried the value of 8, 16, 32. The results showed that both RSNR and SSIM decrease when the patch_size increase, but the runtime and memory drop significantly.
Smaller patches give each token detailed information, so the reconstruction is better. And the limitation is patch_size can't be too small, need to ensure the hardware computational resource before training.
evaluation visualization:
patch_size=8:

patch_size=16:

patch_size=32:

3. Multi-View Geometry with Mini-VGGT (40 points)¶
3.1. Overfitting (10 points)¶
the overfit loss curve image:

GT:¶

predicted geometry:¶

3.2. Training and Evaluation (30 points)¶

| Metric | Val | Test | Better |
|---|---|---|---|
| Depth AbsRel | 0.150 | 0.157 | lower |
| Depth δ1 | 0.809 | 0.791 | higher |
| Rotation AUC @5° / @15° / @30° | 0.190 / 0.535 / 0.710 | 0.223 / 0.550 / 0.741 | higher |
| Translation AUC @5° / @15° / @30° | 0.008 / 0.070 / 0.177 | 0.017 / 0.101 / 0.229 | higher |
| Joint pose AUC @5° / @15° / @30° | 0.003 / 0.053 / 0.154 | 0.003 / 0.077 / 0.211 | higher |
comparing GT and predicted geometry for four different scenes:
scene 09c1414f1b:¶
- GT:

- prediction:

scene 21d970d8de:¶
- GT:

- prediction:

scene 280b83fcf3:¶
- GT:

- prediction:

scene 2b1dc6d6a5:¶
- GT:

- prediction:

The result shows that accuracy improvements in rotation estimation contribute more significantly to the overall pose performance than translation estimation. The reason might be the projection is sensitive to orientation, because the far distance pixel will shift multiple times than the near one.
3.3. Bonus: Analysis and Exploration (10 points)¶
Number of views: How does the number of input views affect accuracy, runtime and memory? Explore changes during training and/or evaluation.
I tested with different number of views, and I haved tried the value=4, 8, 16.
Val:¶
| num_views | Depth AbsRel | Depth δ1 | Rot AUC @5/15/30 | Trans AUC @5/15/30 | Joint AUC@30 |
|---|---|---|---|---|---|
| 4 | 0.129 | 0.849 | 0.213 / 0.562 / 0.715 | 0.004 / 0.061 / 0.165 | 0.140 |
| 8 | 0.150 | 0.809 | 0.190 / 0.535 / 0.710 | 0.008 / 0.070 / 0.177 | 0.154 |
| 16 | 0.179 | 0.769 | 0.183 / 0.487 / 0.683 | 0.009 / 0.063 / 0.164 | 0.137 |
Test:¶
| num_views | Depth AbsRel | Depth δ1 | Rot AUC @5/15/30 | Trans AUC @5/15/30 | Joint AUC@30 |
|---|---|---|---|---|---|
| 4 | 0.145 | 0.819 | 0.261 / 0.573 / 0.759 | 0.013 / 0.106 / 0.216 | 0.191 |
| 8 | 0.157 | 0.791 | 0.223 / 0.550 / 0.741 | 0.017 / 0.101 / 0.229 | 0.211 |
| 16 | 0.169 | 0.764 | 0.228 / 0.557 / 0.741 | 0.013 / 0.097 / 0.232 | 0.215 |
The result shows that depth prediction get worse as num_views grows. The reason might be the model was trained in 8 views. When the num_views increase to 16, there are some views that the model never saw, so the depth prediction drift. Also, as the num_views increases, Rotation angle accuracy drops. The cause might be the severe self-occlusion and viewpoint deformation caused by the larger angles between views.
Evaluating with num_views only indicate the model's generalization to unseen view, not the optimal performance of this model. If we want to know the optimal performance of the num_views actually affect, the training should be done by different num_views then do the evaluation.