Name: Wen Wu
Andrew ID: wengleiw










Canonicalization: Canonicalization is to use a shared coordinate system for 3D points captured from different viewpoints, which each of them has its own camera coordinates. We canonicalize them so no matter the viewpoints of the series of images, they all have a shared anchor to represent relative spatial relationship. During canonicalization, the 3D points from all viewpoints are shared under camera 0's coordinate system. We also normalize by the scale.
World-space ray directions: World-space ray directions means the orientation of the ray from the camera centers to the point in 3D space in world coordinate where the corresponding feature on image is. All rays are expressed in world coordinate.
Plücker moment: Moment is the cross product of ray origins and directions, m = o x d. Two rays may have different origins along the same line, but since they have the same line, they share the same Plücker representation. If two rays share the same line, then o2 = o1 + td, then m2 = o2 x d = (o1 + td) x d = o1 x d + t(d x d) = o1 x d = m1. So Plücker representation can be used to check if two rays are on the same line.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001155954014393501,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Final overfit PSNR (dB): 30.99

Training settings: I trained with 2 source views and 6 target views per sample, 224×224 images, batch size 4, learning rate 2e-4 and 100 epochs. The model uses patch size 8, width 384 and 8 transformer blocks.
Setting: baseline model, 2 source views, correct target rays.
| Source views | Conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.5583 |
| 1 | shuffled | 11.81 | 0.4877 |
| 1 | none | 12.47 | 0.5273 |
| 2 | correct | 16.12 | 0.5601 |
| 2 | shuffled | 11.50 | 0.4723 |
| 2 | none | 12.44 | 0.5267 |
| 4 | correct | 16.87 | 0.5753 |
| 4 | shuffled | 11.59 | 0.4730 |
| 4 | none | 12.45 | 0.5376 |

Setting: baseline model at epoch 95, validation set, 2 source views, correct target rays.
1 source view
1 source view, correct target rays:

1 source view, shuffled target rays:

1 source view, no target rays:

2 source views
2 source views, correct target rays:

2 source views, shuffled target rays:

2 source views, no target rays:

4 source views
4 source views, correct target rays:

4 source views, shuffled target rays:

4 source views, no target rays:

Effect of the number of source views: As the number of source views increases from 1 to 2 and 4, we can see slight increase of PSNR from 15.19 to 16.12 and 16.87. This shows that more views lead to better quality.
Effect of the ray-conditioning ablation: Replacing correct rays with shuffled and none rays both significantly drop PSNR and SSIM. Scores with shuffled rays are below scores with no rays. This shows that the correct target ray is useful for predicting the target view.
Question: How do the patch size, transformer width and transformer depth affect Mini-LVSM's quality, runtime and memory?
Experimental settings: I compared the 2.2 baseline (patch size 8, width 384, 8 transformer blocks) with three variants, each changing one setting: patch size 7, width 768 or 16 transformer blocks. All other settings match 2.2. I evaluated each model's best checkpoint on the validation set with correct target rays and 1, 2 and 4 source views. I also recorded each model's time per epoch and peak GPU memory.
Setting: all models, 2 source views, correct target rays.
| Baseline | patch_size 7 |
|---|---|
![]() |
![]() |
n_layers 16 |
d_model 768 |
|---|---|
![]() |
![]() |
Setting: best checkpoint of each model, validation set, 2 source views, correct target rays.
Results: The figures show validation MSE loss, PSNR and SSIM during training, and the images show predictions for the same validation views (columns: source views, ground truth, prediction). The table compares the four models at 2 source views, the view count used in training.
| Model | PSNR (dB) | SSIM | Runtime (s/epoch) | Peak memory (GiB) |
|---|---|---|---|---|
| Baseline | 16.12 | 0.5601 | 36.3 | 10.4 |
patch_size 7 |
16.16 | 0.5653 | 43.4 | 13.3 |
n_layers 16 |
16.14 | 0.5603 | 56.9 | 18.8 |
d_model 768 |
15.71 | 0.5534 | 64.9 | 19.0 |
PSNR at 1 source view was 15.19 dB for the baseline, 15.19 dB for patch_size 7, 15.16 dB for n_layers 16 and 14.96 dB for d_model 768. At 4 source views, it was 16.87 dB, 16.91 dB, 16.82 dB and 16.17 dB, in the same order. Every model reached its best validation PSNR at the final epoch.
Conclusions: From these results, we can see none of these can visibly improve the image quality. Increasing transformer width even drops the quality, and all experiments use more runtime and memory. There is also no visible improvement in the images. The only tiny increase is using a patch size of 7, which shows some sign of improvement.
Possible explanations and limitations: These results show that the network width and depth do not help with the quality under the same number of epochs, and further reducing patch size may further improve the quality. With no improvement in quality, extra runtime and memory are not what we want. Further reducing patch size may be worth trying, but due to GPU compute limit, 7 is the smallest value I can try.

GT geometry:

Predicted geometry:

Training settings: I trained with the default settings: 8 views per sample, 224×224 images, batch size 16, learning rate 6e-4 and 1000 epochs. The model uses a frozen DINOv2 encoder, 8 pairs of frame and global attention blocks and width 384. The camera loss has weight 5 and the depth loss has weight 1.

Setting: joint training, 8 views.
Setting: best joint model, 8 views.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1409 | 0.1533 |
| Depth δ1 ↑ | 0.8369 | 0.8078 |
| Rotation AUC@5° ↑ | 0.2286 | 0.1964 |
| Rotation AUC@15° ↑ | 0.5689 | 0.5483 |
| Rotation AUC@30° ↑ | 0.7306 | 0.7412 |
| Translation AUC@5° ↑ | 0.0209 | 0.0107 |
| Translation AUC@15° ↑ | 0.1023 | 0.0872 |
| Translation AUC@30° ↑ | 0.2296 | 0.2101 |
| Joint Pose AUC@5° ↑ | 0.0051 | 0.0018 |
| Joint Pose AUC@15° ↑ | 0.0861 | 0.0616 |
| Joint Pose AUC@30° ↑ | 0.2139 | 0.1865 |
Setting: best joint model, validation set, 8 views.
Scene 1 (09c1414f1b), GT geometry:

Scene 1 (09c1414f1b), predicted geometry:

Scene 2 (21d970d8de), GT geometry:

Scene 2 (21d970d8de), predicted geometry:

Scene 3 (280b83fcf3), GT geometry:

Scene 3 (280b83fcf3), predicted geometry:

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 (2b1dc6d6a5), predicted geometry:

Errors on held-out scenes: As we can see from the table above, the error from the testing set is a little worse but very close to the validation set. This applies to almost all metrics except Rotation AUC@30°, showing that our learned model generalizes consistently to unseen scenes. The best-performing metric is Depth δ1, which is above 0.8 for both validation and testing sets.
Question: Does learning cameras and depth jointly improve each task, compared with training each task alone?
Experimental settings: I compared the 3.2 joint model (camera weight 5, depth weight 1) with two single-task models: depth only (camera weight 0) and camera only (depth weight 0). All other settings match 3.2. I evaluated each model's best checkpoint on the validation and test sets.

Setting: joint training, 8 views.

Setting: depth-only training, 8 views.

Setting: camera-only training, 8 views.
Setting: best depth-only model, validation set, 8 views.
Scene 1 (09c1414f1b), GT geometry:

Scene 1 (09c1414f1b), predicted geometry:

Scene 2 (21d970d8de), GT geometry:

Scene 2 (21d970d8de), predicted geometry:

Scene 3 (280b83fcf3), GT geometry:

Scene 3 (280b83fcf3), predicted geometry:

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 (2b1dc6d6a5), predicted geometry:

Setting: best camera-only model, validation set, 8 views.
Scene 1 (09c1414f1b), GT geometry:

Scene 1 (09c1414f1b), predicted geometry:

Scene 2 (21d970d8de), GT geometry:

Scene 2 (21d970d8de), predicted geometry:

Scene 3 (280b83fcf3), GT geometry:

Scene 3 (280b83fcf3), predicted geometry:

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 (2b1dc6d6a5), predicted geometry:

Results: The loss curves show training and validation losses for each run. The screenshots show GT and predicted geometry for the depth-only and camera-only runs on the same four validation scenes as 3.2, which shows the joint model. Camera and depth losses are logged in all runs, including the task with weight 0. The table reports all three models on the same metrics as 3.2.
Setting: best checkpoint of each model, 8 views.
| Metric | Joint, validation | Depth only, validation | Camera only, validation | Joint, test | Depth only, test | Camera only, test |
|---|---|---|---|---|---|---|
| Depth AbsRel ↓ | 0.1409 | 0.1471 | 0.6592 | 0.1533 | 0.1722 | 0.6706 |
| Depth δ1 ↑ | 0.8369 | 0.8128 | 0.2660 | 0.8078 | 0.7690 | 0.2655 |
| Rotation AUC@5° ↑ | 0.2286 | 0.0495 | 0.1566 | 0.1964 | 0.0745 | 0.1977 |
| Rotation AUC@15° ↑ | 0.5689 | 0.1472 | 0.4605 | 0.5483 | 0.1706 | 0.5142 |
| Rotation AUC@30° ↑ | 0.7306 | 0.2860 | 0.6706 | 0.7412 | 0.3256 | 0.7162 |
| Translation AUC@5° ↑ | 0.0209 | 0.0000 | 0.0084 | 0.0107 | 0.0000 | 0.0069 |
| Translation AUC@15° ↑ | 0.1023 | 0.0000 | 0.0572 | 0.0872 | 0.0000 | 0.0484 |
| Translation AUC@30° ↑ | 0.2296 | 0.0000 | 0.1479 | 0.2101 | 0.0000 | 0.1509 |
| Joint Pose AUC@5° ↑ | 0.0051 | 0.0000 | 0.0015 | 0.0018 | 0.0000 | 0.0026 |
| Joint Pose AUC@15° ↑ | 0.0861 | 0.0000 | 0.0436 | 0.0616 | 0.0000 | 0.0332 |
| Joint Pose AUC@30° ↑ | 0.2139 | 0.0000 | 0.1349 | 0.1865 | 0.0000 | 0.1369 |
The depth-only model's camera head and the camera-only model's depth head were not trained, because those tasks have weight 0.
Conclusions: From the table, we can see that learning camera and depth jointly can help both tasks. Compared to depth-only, joint learning improves depth-related metrics on both sets. Compared to camera-only, joint learning improves camera-related metrics on both sets, except for Rotation AUC@5° and Joint Pose AUC@5° on the test set, which show trivial drops.
Possible explanations and limitations: The results show that joint learning may be helpful since the transformer attends to different features describing the same scene. However, our validation and test sets contain a small number of samples. To justify the benefit of joint learning, we need to have more data to test it. In the future, we may also try to incorporate point map prediction to see if it can further advance the results.