Assignment 4 - Avi Athota

1 Multi-View Data Preprocessing

Above is my visualization after unprojection.

Above is my visualization after canonicalization. As you can see, camera 0 is the identity and the centroid distance is 1.

check_data.py output:

{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
    {
    "sequence_name": "e050c15a8d_frame_007080",
    "views": 8,
    "resolution": 224,
    "valid_depth_fraction": 0.9779127240180969,
    "max_reprojection_error_px": 0.00011108566104667261,
    "max_ray_reprojection_error_px": 9.650505671743304e-05,
    "centroid_distance": 1.0
    },
    {
    "sequence_name": "5f99900f09_frame_002820",
    "views": 8,
    "resolution": 224,
    "valid_depth_fraction": 0.9903116226196289,
    "max_reprojection_error_px": 0.00011108566104667261,
    "max_ray_reprojection_error_px": 0.0001078959321603179,
    "centroid_distance": 0.9999999403953552
    },
    {
    "sequence_name": "47b37eb6f9_frame_002720",
    "views": 8,
    "resolution": 224,
    "valid_depth_fraction": 0.9609947800636292,
    "max_reprojection_error_px": 0.00010742284939624369,
    "max_ray_reprojection_error_px": 8.446346328128129e-05,
    "centroid_distance": 1.0
    },
    {
    "sequence_name": "4c5c60fa76_frame_006690",
    "views": 8,
    "resolution": 224,
    "valid_depth_fraction": 0.7516716122627258,
    "max_reprojection_error_px": 0.00010152041068067774,
    "max_ray_reprojection_error_px": 9.860387945082039e-05,
    "centroid_distance": 1.0
    }
],
"batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
    }
}
        

Canonicalization: We move every camera so that it is relative to the first camera, which puts camera 0 at the origin with no rotation. Then, I scaled everything so that the average distance to the scene points is 1. This way, every scene is in the same kind of coordinate frame and also roughly the same size regardless of where it was originally or how big the room is.

World-space ray directions: For each pixel, we get the direction of the ray going through it (by using inverse intrinisics). We then rotate that direction into the world space with the camera's rotation (and finally normalize it). All of the rays from one camera start at that camera's center.

Plucker moment: Each ray is stored as a direction d and a moment m (m = camera center x d). The moment tells you where the line is in space regardless which point on the ray you use to compute it. With this, you are able to describe the ray as a whole line.

2 Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

This is my overfitting visualization (val_epoch000.png). The columns in order are the source views, the GT target, and my prediction. The final PSNR value that I got was 30.86 dB.

2.2 Training and Evaluation

These are my training curves.

ViewsConditionPSNRSSIM
1correct15.210.558
1shuffled11.800.487
1none12.460.529
2correct16.140.560
2shuffled11.480.471
2none12.430.527
4correct16.860.575
4shuffled11.590.473
4none12.450.536

This is a table of my validation PSNR/SSIM for each setting.

 

These are my visualizations at epoch 0 and epoch 95. The columns are the source views, GT, and my prediction.

   

These are my visualizations with correct conditioning using 1, 2, and 4 source views.

 

These are my visualizations with 2 source views using shuffled ray conditioning (left) and no (right) ray conditioning.

My training settings:
- source views: 2
- epochs: 100, batch size: 4
- learning rate: 2e-4

Number of source views: As you can see from the table, more source views gave better results. The PSNR went from 15.21 to 16.14 to 16.86 for 1 to 2 to 4 views. This makes sense because more views shows more of the scene, so the model has less to have to guess.

Ray conditioning: Correct conditioning performed much better than shuffled and no conditioning. No conditioning outputs a blurry guess, since it doesn't know which viewpoint to render. Shuffled is even worse because the model is given the wrong viewpoint.

2.3 Bonus: Analysis and Exploration

For this bonus, I explored what would happen to the evaluation on 1, 2, and 4 source views if I trained with 4 source views instead of 2.

These are the training curves for the model trained with 2 source views vs with 4 source views. Everything else was kept the same as 2.2.

Eval viewsTrained with 2 views PSNR/SSIMTrained with 4 views PSNR/SSIMPSNR change
115.21 / 0.55815.29 / 0.558+0.08
216.14 / 0.56016.39 / 0.562+0.25
416.86 / 0.57517.44 / 0.580+0.58

This is a table of validation PSNR/SSIM with correct conditioning for both models.

 

These are evaluations with 4 source views from the model trained with 2 views and 4 views (right).

Conclusion: Training with 4 source views seemed to improve the results, which was biggest when evaluating with 4 views. The improvement was smaller when evaluating on 2 views, and there was almost no improvement when evaluating on 1 view. This makes logical sense because training the model with more views teaches it how to combine info from several images, and it does best when tested in the same way that it was trained. The drawback to this is that each epoch took slightly longer (29 seconds for 4 views, 25 seconds for 2 views) because there is more to process.

3 Multi-View Geometry with Mini-VGGT

3.1 Overfitting

These are my overfitting loss curves.

 

These are screenshots of the GT geometry on the left and my prediction on the right. As you can see, they are pretty close, though the prediction is a little bit blurrier/noisier.

3.2 Training and Evaluation

These are my training curves.

SplitAbsRelδ1Rotation AUC @5/15/30Translation AUC @5/15/30Pose AUC @5/15/30
val0.1560.8070.201 / 0.530 / 0.7070.021 / 0.094 / 0.2000.008 / 0.074 / 0.179
test0.1460.8090.207 / 0.557 / 0.7520.014 / 0.076 / 0.1830.002 / 0.056 / 0.169

This is a table of my validation and test metrics.

 

 

 

 

These are my GT vs prediction on the 4 different scenes.

My training settings:
- views: 8, sampling range: 32
- epochs: 1000, batch size: 16
- learning rate: 6e-4

Errors on held-out scenes: From the table, the model seems to do a decent job on depth, and the val and test numbers seem pretty close. But compared to 3.1, the prediction images are much more blurrier and inaccurate. You can see from both the table and images that the biggest error seems to be in the camera translation (especially in scene 4, where the camera barely moved).

3.3 Bonus: Analysis and Exploration

For this bonus, I explored how the number of input views affects accuracy, runtime, and memory. To do this I compared my model from 3.2 (which was trained with 8 views) to a second model I trained with 4 views. This was the only thing I changed from 3.2.

These are the training curves for the model trained with 4 views.

Train viewsEval viewsAbsRelδ1Rotation AUC@30Pose AUC@30
840.1400.8310.6960.180
880.1560.8070.7070.179
8160.1760.7860.6810.156
440.1300.8470.6900.129
480.1470.8090.6900.133
4160.1720.7840.6620.125

This is a table of validation accuracy for both models with varying numbers of eval views.

Views2481632
Time per sample (ms)3232323770
Peak GPU memory (MB)7077247718511001

This is the runtime and memory of a single forward pass for varying number of views. The total train times were ~5 hours for 8 views and ~3 hours for 4 views.

Conclusion: More eval views made depth worse for both of the models. THe pose dropped at 16 views, which is more than either model was trained on. The 4 view model had slightly better depth but much worse pose than the 8 view model. This is likely because the larger number of views gives it more camera pairs to learn from. The forward pass runtimes were about the same up to 8 views, but the training part for 8 views took almost 70% longer. I would say the main limitation of this experiment is that I only trained each model once so its possible that the outputs are a little noisy.