16-825 Assignment 4: Multi-View Transformers

Name: Wen Wu
Andrew ID: wengleiw

1. Multi-View Data Preprocessing

1.2 Depth Unprojection

Unprojected points and cameras in the world frame, scene 1

Unprojected points and cameras in the world frame, scene 2

Unprojected points and cameras in the world frame, scene 3

Unprojected points and cameras in the world frame, scene 4

Unprojected points and cameras in the world frame, scene 5

1.3 Camera and Depth Canonicalization

Canonicalized points and cameras, scene 1

Canonicalized points and cameras, scene 2

Canonicalized points and cameras, scene 3

Canonicalized points and cameras, scene 4

Canonicalized points and cameras, scene 5

Canonicalization: Canonicalization is to use a shared coordinate system for 3D points captured from different viewpoints, which each of them has its own camera coordinates. We canonicalize them so no matter the viewpoints of the series of images, they all have a shared anchor to represent relative spatial relationship. During canonicalization, the 3D points from all viewpoints are shared under camera 0's coordinate system. We also normalize by the scale.

1.4 Camera Rays

World-space ray directions: World-space ray directions means the orientation of the ray from the camera centers to the point in 3D space in world coordinate where the corresponding feature on image is. All rays are expressed in world coordinate.

1.5 Plücker Coordinates

Plücker moment: Moment is the cross product of ray origins and directions, m = o x d. Two rays may have different origins along the same line, but since they have the same line, they share the same Plücker representation. If two rays share the same line, then o2 = o1 + td, then m2 = o2 x d = (o1 + td) x d = o1 x d + t(d x d) = o1 x d = m1. So Plücker representation can be used to check if two rays are on the same line.

1.7 Final Check

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR (dB): 30.99

Overfit visualization

2.2 Training and Evaluation

Training settings: I trained with 2 source views and 6 target views per sample, 224×224 images, batch size 4, learning rate 2e-4 and 100 epochs. The model uses patch size 8, width 384 and 8 transformer blocks.

LVSM MSE loss LVSM PSNR LVSM SSIM

Setting: baseline model, 2 source views, correct target rays.

Source views Conditioning PSNR (dB) SSIM
1 correct 15.19 0.5583
1 shuffled 11.81 0.4877
1 none 12.47 0.5273
2 correct 16.12 0.5601
2 shuffled 11.50 0.4723
2 none 12.44 0.5267
4 correct 16.87 0.5753
4 shuffled 11.59 0.4730
4 none 12.45 0.5376

Training Visualization

LVSM training visualization

Setting: baseline model at epoch 95, validation set, 2 source views, correct target rays.

Evaluation Visualizations

1 source view

1 source view, correct target rays:

1 source view, correct rays

1 source view, shuffled target rays:

1 source view, shuffled rays

1 source view, no target rays:

1 source view, no rays

2 source views

2 source views, correct target rays:

2 source views, correct rays

2 source views, shuffled target rays:

2 source views, shuffled rays

2 source views, no target rays:

2 source views, no rays

4 source views

4 source views, correct target rays:

4 source views, correct rays

4 source views, shuffled target rays:

4 source views, shuffled rays

4 source views, no target rays:

4 source views, no rays

Effect of the number of source views: As the number of source views increases from 1 to 2 and 4, we can see slight increase of PSNR from 15.19 to 16.12 and 16.87. This shows that more views lead to better quality.

Effect of the ray-conditioning ablation: Replacing correct rays with shuffled and none rays both significantly drop PSNR and SSIM. Scores with shuffled rays are below scores with no rays. This shows that the correct target ray is useful for predicting the target view.

2.3 Bonus: Analysis and Exploration

Question: How do the patch size, transformer width and transformer depth affect Mini-LVSM's quality, runtime and memory?

Experimental settings: I compared the 2.2 baseline (patch size 8, width 384, 8 transformer blocks) with three variants, each changing one setting: patch size 7, width 768 or 16 transformer blocks. All other settings match 2.2. I evaluated each model's best checkpoint on the validation set with correct target rays and 1, 2 and 4 source views. I also recorded each model's time per epoch and peak GPU memory.

LVSM exploration MSE loss LVSM exploration PSNR LVSM exploration SSIM

Setting: all models, 2 source views, correct target rays.

Baseline patch_size 7
Baseline patch_size 7
n_layers 16 d_model 768
n_layers 16 d_model 768

Setting: best checkpoint of each model, validation set, 2 source views, correct target rays.

Results: The figures show validation MSE loss, PSNR and SSIM during training, and the images show predictions for the same validation views (columns: source views, ground truth, prediction). The table compares the four models at 2 source views, the view count used in training.

Model PSNR (dB) SSIM Runtime (s/epoch) Peak memory (GiB)
Baseline 16.12 0.5601 36.3 10.4
patch_size 7 16.16 0.5653 43.4 13.3
n_layers 16 16.14 0.5603 56.9 18.8
d_model 768 15.71 0.5534 64.9 19.0

PSNR at 1 source view was 15.19 dB for the baseline, 15.19 dB for patch_size 7, 15.16 dB for n_layers 16 and 14.96 dB for d_model 768. At 4 source views, it was 16.87 dB, 16.91 dB, 16.82 dB and 16.17 dB, in the same order. Every model reached its best validation PSNR at the final epoch.

Conclusions: From these results, we can see none of these can visibly improve the image quality. Increasing transformer width even drops the quality, and all experiments use more runtime and memory. There is also no visible improvement in the images. The only tiny increase is using a patch size of 7, which shows some sign of improvement.

Possible explanations and limitations: These results show that the network width and depth do not help with the quality under the same number of epochs, and further reducing patch size may further improve the quality. With no improvement in quality, extra runtime and memory are not what we want. Further reducing patch size may be worth trying, but due to GPU compute limit, 7 is the smallest value I can try.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit loss curve (total, camera, depth)

GT geometry:

Overfit GT geometry

Predicted geometry:

Overfit predicted geometry

3.2 Training and Evaluation

Training settings: I trained with the default settings: 8 views per sample, 224×224 images, batch size 16, learning rate 6e-4 and 1000 epochs. The model uses a frozen DINOv2 encoder, 8 pairs of frame and global attention blocks and width 384. The camera loss has weight 5 and the depth loss has weight 1.

Training loss curve (total, camera, depth)

Setting: joint training, 8 views.

Setting: best joint model, 8 views.

Metric Validation Test
Depth AbsRel ↓ 0.1409 0.1533
Depth δ1 ↑ 0.8369 0.8078
Rotation AUC@5° ↑ 0.2286 0.1964
Rotation AUC@15° ↑ 0.5689 0.5483
Rotation AUC@30° ↑ 0.7306 0.7412
Translation AUC@5° ↑ 0.0209 0.0107
Translation AUC@15° ↑ 0.1023 0.0872
Translation AUC@30° ↑ 0.2296 0.2101
Joint Pose AUC@5° ↑ 0.0051 0.0018
Joint Pose AUC@15° ↑ 0.0861 0.0616
Joint Pose AUC@30° ↑ 0.2139 0.1865

Setting: best joint model, validation set, 8 views.

Scene 1 (09c1414f1b), GT geometry:

Scene 1 GT geometry

Scene 1 (09c1414f1b), predicted geometry:

Scene 1 predicted geometry

Scene 2 (21d970d8de), GT geometry:

Scene 2 GT geometry

Scene 2 (21d970d8de), predicted geometry:

Scene 2 predicted geometry

Scene 3 (280b83fcf3), GT geometry:

Scene 3 GT geometry

Scene 3 (280b83fcf3), predicted geometry:

Scene 3 predicted geometry

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 GT geometry

Scene 4 (2b1dc6d6a5), predicted geometry:

Scene 4 predicted geometry

Errors on held-out scenes: As we can see from the table above, the error from the testing set is a little worse but very close to the validation set. This applies to almost all metrics except Rotation AUC@30°, showing that our learned model generalizes consistently to unseen scenes. The best-performing metric is Depth δ1, which is above 0.8 for both validation and testing sets.

3.3 Bonus: Analysis and Exploration

Question: Does learning cameras and depth jointly improve each task, compared with training each task alone?

Experimental settings: I compared the 3.2 joint model (camera weight 5, depth weight 1) with two single-task models: depth only (camera weight 0) and camera only (depth weight 0). All other settings match 3.2. I evaluated each model's best checkpoint on the validation and test sets.

Joint training loss curve (total, camera, depth)

Setting: joint training, 8 views.

Depth-only loss curve (total, camera, depth)

Setting: depth-only training, 8 views.

Camera-only loss curve (total, camera, depth)

Setting: camera-only training, 8 views.

Setting: best depth-only model, validation set, 8 views.

Scene 1 (09c1414f1b), GT geometry:

Scene 1 GT geometry, depth only

Scene 1 (09c1414f1b), predicted geometry:

Scene 1 predicted geometry, depth only

Scene 2 (21d970d8de), GT geometry:

Scene 2 GT geometry, depth only

Scene 2 (21d970d8de), predicted geometry:

Scene 2 predicted geometry, depth only

Scene 3 (280b83fcf3), GT geometry:

Scene 3 GT geometry, depth only

Scene 3 (280b83fcf3), predicted geometry:

Scene 3 predicted geometry, depth only

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 GT geometry, depth only

Scene 4 (2b1dc6d6a5), predicted geometry:

Scene 4 predicted geometry, depth only

Setting: best camera-only model, validation set, 8 views.

Scene 1 (09c1414f1b), GT geometry:

Scene 1 GT geometry, camera only

Scene 1 (09c1414f1b), predicted geometry:

Scene 1 predicted geometry, camera only

Scene 2 (21d970d8de), GT geometry:

Scene 2 GT geometry, camera only

Scene 2 (21d970d8de), predicted geometry:

Scene 2 predicted geometry, camera only

Scene 3 (280b83fcf3), GT geometry:

Scene 3 GT geometry, camera only

Scene 3 (280b83fcf3), predicted geometry:

Scene 3 predicted geometry, camera only

Scene 4 (2b1dc6d6a5), GT geometry:

Scene 4 GT geometry, camera only

Scene 4 (2b1dc6d6a5), predicted geometry:

Scene 4 predicted geometry, camera only

Results: The loss curves show training and validation losses for each run. The screenshots show GT and predicted geometry for the depth-only and camera-only runs on the same four validation scenes as 3.2, which shows the joint model. Camera and depth losses are logged in all runs, including the task with weight 0. The table reports all three models on the same metrics as 3.2.

Setting: best checkpoint of each model, 8 views.

Metric Joint, validation Depth only, validation Camera only, validation Joint, test Depth only, test Camera only, test
Depth AbsRel ↓ 0.1409 0.1471 0.6592 0.1533 0.1722 0.6706
Depth δ1 ↑ 0.8369 0.8128 0.2660 0.8078 0.7690 0.2655
Rotation AUC@5° ↑ 0.2286 0.0495 0.1566 0.1964 0.0745 0.1977
Rotation AUC@15° ↑ 0.5689 0.1472 0.4605 0.5483 0.1706 0.5142
Rotation AUC@30° ↑ 0.7306 0.2860 0.6706 0.7412 0.3256 0.7162
Translation AUC@5° ↑ 0.0209 0.0000 0.0084 0.0107 0.0000 0.0069
Translation AUC@15° ↑ 0.1023 0.0000 0.0572 0.0872 0.0000 0.0484
Translation AUC@30° ↑ 0.2296 0.0000 0.1479 0.2101 0.0000 0.1509
Joint Pose AUC@5° ↑ 0.0051 0.0000 0.0015 0.0018 0.0000 0.0026
Joint Pose AUC@15° ↑ 0.0861 0.0000 0.0436 0.0616 0.0000 0.0332
Joint Pose AUC@30° ↑ 0.2139 0.0000 0.1349 0.1865 0.0000 0.1369

The depth-only model's camera head and the camera-only model's depth head were not trained, because those tasks have weight 0.

Conclusions: From the table, we can see that learning camera and depth jointly can help both tasks. Compared to depth-only, joint learning improves depth-related metrics on both sets. Compared to camera-only, joint learning improves camera-related metrics on both sets, except for Rotation AUC@5° and Joint Pose AUC@5° on the test set, which show trivial drops.

Possible explanations and limitations: The results show that joint learning may be helpful since the transformer attends to different features describing the same scene. However, our validation and test sets contain a small number of samples. To justify the benefit of joint learning, we need to have more data to test it. In the future, we may also try to incorporate point map prediction to see if it can further advance the results.