3D Vision Assignment 4: Multi-View Transformers

Q1: Multi-View Data Preprocessing

Depth Unprojection & Canonicalization

Scene 1

Depth Unprojection
Canonicalization
Centroid Information

Scene 2

Depth Unprojection
Canonicalization
Centroid Information

Scene 3

Depth Unprojection
Canonicalization
Centroid Information

Scene 4

Depth Unprojection
Canonicalization
Centroid Information

Scene 5

Depth Unprojection
Canonicalization
Centroid Information


Final Check

                    {
                        "split": "train",
                        "dataset_samples": 224,
                        "checked_samples": 4,
                        "samples": [
                            {
                            "sequence_name": "e050c15a8d_frame_007080",
                            "views": 8,
                            "resolution": 224,
                            "valid_depth_fraction": 0.9779127240180969,
                            "max_reprojection_error_px": 0.00011009880108758807,
                            "max_ray_reprojection_error_px": 9.394258086103946e-05,
                            "centroid_distance": 1.0
                            },
                            {
                            "sequence_name": "5f99900f09_frame_002820",
                            "views": 8,
                            "resolution": 224,
                            "valid_depth_fraction": 0.9903116226196289,
                            "max_reprojection_error_px": 0.00011108566104667261,
                            "max_ray_reprojection_error_px": 0.00011108566104667261,
                            "centroid_distance": 0.9999999403953552
                            },
                            {
                            "sequence_name": "47b37eb6f9_frame_002720",
                            "views": 8,
                            "resolution": 224,
                            "valid_depth_fraction": 0.9609947800636292,
                            "max_reprojection_error_px": 9.351845073979348e-05,
                            "max_ray_reprojection_error_px": 8.897326188161969e-05,
                            "centroid_distance": 1.0
                            },
                            {
                            "sequence_name": "4c5c60fa76_frame_006690",
                            "views": 8,
                            "resolution": 224,
                            "valid_depth_fraction": 0.7516716122627258,
                            "max_reprojection_error_px": 0.00010746094631031156,
                            "max_ray_reprojection_error_px": 9.423537994734943e-05,
                            "centroid_distance": 0.9999999403953552
                            }
                        ],
                        "batch_shapes": {
                            "rgb": [
                            2,
                            8,
                            3,
                            224,
                            224
                            ],
                            "depth": [
                            2,
                            8,
                            1,
                            224,
                            224
                            ],
                            "depth_valid": [
                            2,
                            8,
                            1,
                            224,
                            224
                            ],
                            "K": [
                            2,
                            8,
                            3,
                            3
                            ],
                            "frame_numbers": [
                            2,
                            8
                            ],
                            "T_world_from_camera": [
                            2,
                            8,
                            4,
                            4
                            ],
                            "original_world_from_reference": [
                            2,
                            4,
                            4
                            ],
                            "scene_scale": [
                            2
                            ],
                            "points_world": [
                            2,
                            8,
                            3,
                            224,
                            224
                            ],
                            "ray_origins": [
                            2,
                            8,
                            3,
                            224,
                            224
                            ],
                            "ray_directions": [
                            2,
                            8,
                            3,
                            224,
                            224
                            ],
                            "plucker": [
                            2,
                            8,
                            6,
                            224,
                            224
                            ]
                        }
                    }
                

In canonicalization, we put every scene in a consistent coordinate system, selected as camera 0. Its pose becomes the reference pose for all other camera poses (i.e. it becomes the identity). We then express all other camera poses and depths relative to that reference and scale the scene accordingly.
World-space ray directions are the directions in the shared 3D world space that pixels look towards. A ray is created from the pixel into the 3D scene.
The Plücker moment is a representation of the ray's direction and position relative to the origin, formed by m = o x d.

Q2: Novel-View Synthesis with Mini-LVSM

Overfitting

Train Loss: 0.00513
Train PSNR: 29.14
Validation Loss: 0.00021
Validation PSNR: 36.83

Validation Epoch 0

Training and Evaluation

Training Curves for LVSM

Source views Target-ray conditioning PSNR SSIM N
1Correct13.970.546196
1Shuffled12.110.516196
1None (zero rays)12.200.544196
2Correct14.290.544168
2Shuffled12.050.510168
2None (zero rays)12.140.540168
4Correct14.440.554112
4Shuffled12.190.517112
4None (zero rays)12.140.550112
Training Validation Example: Epoch 5
Training Validation Example: Epoch 95

1 Source View (Correct)
1 Source View (Shuffled)
1 Source View (All-Zero)

2 Source Views (Correct)
2 Source Views (Shuffled)
2 Source Views (All-Zero)

4 Source Views (Correct)
4 Source Views (Shuffled)
4 Source Views (All-Zero)

Settings:
I used the default settings with some adjustments due to hardware limitations. The program ran for 100 epochs with 2 source views and 8 total views per sample at 224x224 resolution. I increased the patch size from 8 to 16 to reduce the token size and decreased the batch size to 1.

Increasing the number of source views generally improved image quality. As the number of source views rose in the ablations, the PSNR also rose from 13.97 dB with one source to 14.44 dB with four sources. SSIM was also marginally higher with four sources. Correct target rays also gave higher PSNR than either zero or shuffled target rays, which implies that incorrect target-camera information will harm the prediction similarity. While the final results came out blocky and lacking detail, this run was only for 100 epochs due to time and hardware limitations; as the PSNR and SSIM values rose over epochs according to the training curves and final metrics, more epochs likely would have yielded a sharper image.

Analysis and Exploration

I wanted to experiment with the number of source views as my computer hardware would allow. I tried lowering the number of source views from 2 to 1 to see how it would affect the quality of results. As expected, having less source views to train with caused a worse prediction:

1 Source View (Correct)
1 Source View (Shuffled)
1 Source View (All-Zero)

2 Source Views (Correct)
2 Source Views (Shuffled)
2 Source Views (All-Zero)

4 Source Views (Correct)
4 Source Views (Shuffled)
4 Source Views (All-Zero)

PSNR/SSIM comparison between runs
Source views Target-ray conditioning 1-View PSNR 1-View SSIM 2-View PSNR 2-View SSIM Δ PSNR Δ SSIM
1Correct13.9850.545313.9690.5459−0.015+0.0007
1Shuffled12.0240.514412.1120.5161+0.088+0.0016
1None12.3420.541312.2030.5437−0.138+0.0025
2Correct13.9390.543114.2880.5445+0.348+0.0014
2Shuffled12.0830.513712.0500.5104−0.033−0.0033
2None12.3550.538212.1360.5398−0.219+0.0016
4Correct13.6590.538114.4360.5536+0.777+0.0154
4Shuffled12.4440.517412.1860.5171−0.258−0.0003
4None12.3200.536512.1430.5500−0.177+0.0135

Due to hardware limitations, I was not able to experiment with increasing the number of source views beyond what was initially trained with, but these results suggest that there is an increase in quality that comes from the number of source views as there is more information available to the model to construct a novel view with. As shown by the table, there are mostly increases in the similarity metrics when trained with more source views.

Q3: Multi-View Geometry with Mini-VGGT

Overfitting

Overfit Loss Curve

Ground Truth Geometry
Predicted Geometry

Training and Evaluation

Training Curves for VGGT

Split Depth Rotation AUC Translation AUC Joint Pose AUC
AbsRel δ1 5° 15° 30° 5° 15° 30° 5° 15° 30°
Validation 0.1550 0.7901 0.1360 0.3974 0.6009 0.0059 0.0420 0.1111 0.0018 0.0270 0.0947
Test 0.1731 0.7316 0.1286 0.4259 0.6618 0.0077 0.0359 0.1146 0.0010 0.0211 0.0988
Ground Truth Geometry
Predicted Geometry

Ground Truth Geometry
Predicted Geometry

Ground Truth Geometry
Predicted Geometry

Ground Truth Geometry
Predicted Geometry

Settings:
I used the default settings with some adjustments due to hardware limitations. The program ran for 232 epochs with a batch size of 2 and 1 worker. Unfortunately, any changes to these settings proved too taxing for the computer and even this run was not able to complete the full 1000 epochs. Provided are the best results for the last working checkpoint.

The results still approach the geometry displayed by the ground-truth views for each scene, suggesting that the full number of epochs would have achieved a significantly higher-quality prediction. Depth generalization is reasonable according to the metrics, though not high; this may improve over more training epochs but also shows a drop between splits. Rotation performance is stronger in test but overall still quite inaccurate, while translation accuracy is extremely weak across both splits.
Overall, depth is often in the right range but still has a somewhat high amount of error, as only about 73-79% of pixels meet the δ1 threshold. Meanwhile, rotation error is moderate but not extremely large; still, the low 5° AUC implies that very accurate rotations are not common. Translation has a very high amount of error in all metrics. The training loss curves show a gradual decrease in loss over epochs, which shows that more training might have an impact on the quality of the prediction.