

Above is my visualization after unprojection.


Above is my visualization after canonicalization. As you can see, camera 0 is the identity and the centroid distance is 1.
check_data.py output:
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 9.650505671743304e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00010742284939624369,
"max_ray_reprojection_error_px": 8.446346328128129e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010152041068067774,
"max_ray_reprojection_error_px": 9.860387945082039e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Canonicalization: We move every camera so that it is relative to the first camera, which puts camera 0 at the origin with no rotation. Then, I scaled everything so that the average distance to the scene points is 1. This way, every scene is in the same kind of coordinate frame and also roughly the same size regardless of where it was originally or how big the room is.
World-space ray directions: For each pixel, we get the direction of the ray going through it (by using inverse intrinisics). We then rotate that direction into the world space with the camera's rotation (and finally normalize it). All of the rays from one camera start at that camera's center.
Plucker moment: Each ray is stored as a direction d and a moment m (m = camera center x d). The moment tells you where the line is in space regardless which point on the ray you use to compute it. With this, you are able to describe the ray as a whole line.

This is my overfitting visualization (val_epoch000.png). The columns in order are the source views, the GT target, and my prediction. The final PSNR value that I got was 30.86 dB.

These are my training curves.
| Views | Condition | PSNR | SSIM |
|---|---|---|---|
| 1 | correct | 15.21 | 0.558 |
| 1 | shuffled | 11.80 | 0.487 |
| 1 | none | 12.46 | 0.529 |
| 2 | correct | 16.14 | 0.560 |
| 2 | shuffled | 11.48 | 0.471 |
| 2 | none | 12.43 | 0.527 |
| 4 | correct | 16.86 | 0.575 |
| 4 | shuffled | 11.59 | 0.473 |
| 4 | none | 12.45 | 0.536 |
This is a table of my validation PSNR/SSIM for each setting.

These are my visualizations at epoch 0 and epoch 95. The columns are the source views, GT, and my prediction.

These are my visualizations with correct conditioning using 1, 2, and 4 source views.

These are my visualizations with 2 source views using shuffled ray conditioning (left) and no (right) ray conditioning.
My training settings:
- source views: 2
- epochs: 100, batch size: 4
- learning rate: 2e-4
Number of source views: As you can see from the table, more source views gave better results. The PSNR went from 15.21 to 16.14 to 16.86 for 1 to 2 to 4 views. This makes sense because more views shows more of the scene, so the model has less to have to guess.
Ray conditioning: Correct conditioning performed much better than shuffled and no conditioning. No conditioning outputs a blurry guess, since it doesn't know which viewpoint to render. Shuffled is even worse because the model is given the wrong viewpoint.
For this bonus, I explored what would happen to the evaluation on 1, 2, and 4 source views if I trained with 4 source views instead of 2.

These are the training curves for the model trained with 2 source views vs with 4 source views. Everything else was kept the same as 2.2.
| Eval views | Trained with 2 views PSNR/SSIM | Trained with 4 views PSNR/SSIM | PSNR change |
|---|---|---|---|
| 1 | 15.21 / 0.558 | 15.29 / 0.558 | +0.08 |
| 2 | 16.14 / 0.560 | 16.39 / 0.562 | +0.25 |
| 4 | 16.86 / 0.575 | 17.44 / 0.580 | +0.58 |
This is a table of validation PSNR/SSIM with correct conditioning for both models.

These are evaluations with 4 source views from the model trained with 2 views and 4 views (right).
Conclusion: Training with 4 source views seemed to improve the results, which was biggest when evaluating with 4 views. The improvement was smaller when evaluating on 2 views, and there was almost no improvement when evaluating on 1 view. This makes logical sense because training the model with more views teaches it how to combine info from several images, and it does best when tested in the same way that it was trained. The drawback to this is that each epoch took slightly longer (29 seconds for 4 views, 25 seconds for 2 views) because there is more to process.

These are my overfitting loss curves.

These are screenshots of the GT geometry on the left and my prediction on the right. As you can see, they are pretty close, though the prediction is a little bit blurrier/noisier.

These are my training curves.
| Split | AbsRel | δ1 | Rotation AUC @5/15/30 | Translation AUC @5/15/30 | Pose AUC @5/15/30 |
|---|---|---|---|---|---|
| val | 0.156 | 0.807 | 0.201 / 0.530 / 0.707 | 0.021 / 0.094 / 0.200 | 0.008 / 0.074 / 0.179 |
| test | 0.146 | 0.809 | 0.207 / 0.557 / 0.752 | 0.014 / 0.076 / 0.183 | 0.002 / 0.056 / 0.169 |
This is a table of my validation and test metrics.




These are my GT vs prediction on the 4 different scenes.
My training settings:
- views: 8, sampling range: 32
- epochs: 1000, batch size: 16
- learning rate: 6e-4
Errors on held-out scenes: From the table, the model seems to do a decent job on depth, and the val and test numbers seem pretty close. But compared to 3.1, the prediction images are much more blurrier and inaccurate. You can see from both the table and images that the biggest error seems to be in the camera translation (especially in scene 4, where the camera barely moved).
For this bonus, I explored how the number of input views affects accuracy, runtime, and memory. To do this I compared my model from 3.2 (which was trained with 8 views) to a second model I trained with 4 views. This was the only thing I changed from 3.2.

These are the training curves for the model trained with 4 views.

| Train views | Eval views | AbsRel | δ1 | Rotation AUC@30 | Pose AUC@30 |
|---|---|---|---|---|---|
| 8 | 4 | 0.140 | 0.831 | 0.696 | 0.180 |
| 8 | 8 | 0.156 | 0.807 | 0.707 | 0.179 |
| 8 | 16 | 0.176 | 0.786 | 0.681 | 0.156 |
| 4 | 4 | 0.130 | 0.847 | 0.690 | 0.129 |
| 4 | 8 | 0.147 | 0.809 | 0.690 | 0.133 |
| 4 | 16 | 0.172 | 0.784 | 0.662 | 0.125 |
This is a table of validation accuracy for both models with varying numbers of eval views.

| Views | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|
| Time per sample (ms) | 32 | 32 | 32 | 37 | 70 |
| Peak GPU memory (MB) | 707 | 724 | 771 | 851 | 1001 |
This is the runtime and memory of a single forward pass for varying number of views. The total train times were ~5 hours for 8 views and ~3 hours for 4 views.
Conclusion: More eval views made depth worse for both of the models. THe pose dropped at 16 views, which is more than either model was trained on. The 4 view model had slightly better depth but much worse pose than the 8 view model. This is likely because the larger number of views gives it more camera pairs to learn from. The forward pass runtimes were about the same up to 8 views, but the training part for 8 views took almost 70% longer. I would say the main limitation of this experiment is that I only trained each model once so its possible that the outputs are a little noisy.