16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Data visualization screenshots

The following results show the same sampled views after unprojection and canonicalization. The first sampled camera is shown in red. Each screenshot of the canonicalization visualization includes the first camera center and the valid-point centroid distance in the sidebar; all three centers are approximately (0, 0, 0) and all three centroid distances are 1.000.

Scene: 079a326597_frame_007430

Unprojection (original world coordinates, m)

Unprojection: 079a326597_frame_007430

Canonicalization (first-camera coordinates, normalized units)

Canonicalization: 079a326597_frame_007430

Scene: 07f5b601ee_frame_003630

Unprojection (original world coordinates, m)

Unprojection: 07f5b601ee_frame_003630

Canonicalization (first-camera coordinates, normalized units)

Canonicalization: 07f5b601ee_frame_003630

Scene: 076c822ecc_frame_002890

Unprojection (original world coordinates, m)

Unprojection: 076c822ecc_frame_002890

Canonicalization (first-camera coordinates, normalized units)

Canonicalization: 076c822ecc_frame_002890

Click any screenshot to view its numerical annotations at full resolution. Missing regions and scattered points reflect incomplete or noisy input depth.

check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

Across the four checked samples, the reported tensor shapes match the data contract, the maximum point and ray reprojection errors are approximately 0.000111 pixels (< 0.01 pixels), and the valid-point centroid distances are approximately 1.0,

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.351845073979348e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010400763130746782,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization is the transformation of multi-view geometry into the first camera's coordinate frame (The first camera has the identity pose). A shared scale makes the valid-point centroid distance equal to 1.

World-space ray direction is the unit ray direction (unit vector pointing from a camera center through a pixel), expressed in the world coordinate frame. It is computed as d = normalize(R @ inverse(K) @ [u, v, 1]).

Plücker moment is the vector m = o × d, where o is the world-space ray origin in canonical units and d is the unit world-space ray direction. Together, [d, m] encodes the oriented line’s direction and position.

2. Novel-View Synthesis with Mini-LVSM

2.1. Overfitting

The final overfit PSNR was 30.89 dB , which reaches about 30 dB or more.

Mini-LVSM overfit visualization

Each row shows (from left to right): two source views, the ground-truth target, and the prediction.

2.2. Training and Evaluation

Mini-LVSM Training settings.

Training curves

curves

Validation ablations

All settings below use the same best checkpoint which is saved at epoch 95 (counted from epoch 0)

Number of source views Target-ray conditioning PSNR (dB) SSIM Target images
1 correct 15.2192 0.5573 196
1 shuffled 11.7826 0.4862 196
1 none 12.4361 0.5287 196
2 correct 16.1787 0.5594 168
2 shuffled 11.4341 0.4707 168
2 none 12.4098 0.5257 168
4 correct 16.8570 0.5737 112
4 shuffled 11.5636 0.4739 112
4 none 12.4629 0.5362 112

Training Visualizations

Each row shows (from left to right): two source views, the ground-truth target, and the prediction.

Epoch 1:

Training epoch 1

Epoch 51:

Training epoch 51

Epoch 96:

Training epoch 96

Evaluation visualizations

Each row shows (from left to right): the source views (in the first 1, 2, or 4 columns), the ground-truth target, and the prediction.

1 source view, correct target rays:

1 sources - correct

1 source view, shuffled target rays:

1 sources - shuffled

1 source view, none target rays:

1 sources - none

2 source views, correct target rays:

2 sources - correct

2 source views, shuffled target rays:

2 sources - shuffled

2 source views, none target rays:

2 sources - none

4 source views, correct target rays:

4 sources - correct

4 source views, shuffled target rays:

4 sources, shuffled

4 source views, none target rays:

4 sources - none

Discussion: number of source views

With correct target rays, increasing the number of source views from 1 to 2 to 4 improves both PSNR and SSIM. PSNR increases from 15.22 to 16.18 to 16.86 dB, while SSIM increases from 0.5573 to 0.5594 to 0.5737. More source views provide additional scene information and help better constrain the target-view prediction.

Discussion: ray conditioning

Among experiments with the same number of source views, correct target rays achieve the best performance, while shuffled target rays yield the worst performance. For instance, with 4 source views, shuffling the target rays reduces PSNR from 16.86 to 11.56 dB and SSIM from 0.5737 to 0.4739. Removing the target rays reduces PSNR to 12.46 dB and SSIM to 0.5362, therefore experiments without target rays perform between those with correct and shuffled rays. This suggests that incorrect target rays can be more harmful than missing target rays, showing that correct ray conditioning is crucial for reconstruction.

2.3. Bonus: Analysis and Exploration

Question: How does adding an L1 term to the MSE loss affect PSNR, SSIM, and visual sharpness?

Experimental settings: The training objective is L = MSE + lambda * L1, with lambda = 0, 0.1, 0.5, 1.0. The MSE baseline from Section 2.2 is the experiment run of lambda = 0. Each model is trained from scratch for 100 epochs with the same model architecture, data settings, seed and optimizer as Section 2.2. The best checkpoint is selected by validation PSNR and evaluated on the same validation scenes using 2 source views and correct target rays.

Results:

L1 weight Validation PSNR (dB) Validation SSIM
0 (Single MSE loss baseline) 16.1787 0.5594
0.1 16.2569 0.5604
0.5 16.3368 0.5633
1.0 16.3265 0.5640

All three L1 settings very slightly improve PSNR and SSIM over the baseline. A weight of 0.5 achieves the highest PSNR (16.3368 dB). A weight of 1.0 achieves the highest SSIM (0.5640).

The following images show the same source and target views across all four loss settings. Each row contains two source views, the ground-truth target, and the prediction. The predictions largely remain blurry and the visual improvement is limited.

L1 weight = 0:

L1 weight 0

L1 weight = 0.1:

L1 weight 0.1

L1 weight = 0.5:

L1 weight 0.5

L1 weight = 1.0:

L1 weight 1.0

Possible explanation: Adding L1 loss gives smaller pixel errors more weight relative to using only the MSE loss. Adding L1 loss can change how the model balances errors across pixels. This may lead to the small gains in metrics. However, both losses operate on individual pixels and do not explicitly enforce sharp edges or enhance visual details.

Limitations: Each setting uses only one seed, so the small differences may reflect training variability. The visual predictions suggest that the model is still underfitting.

Conclusion: Adding L1 provides a small improvement in validation PSNR and SSIM for these runs. L1 alone does not visibly improve the blurry, blocky predictions.

3. Multi-View Geometry with Mini-VGGT

3.1. Overfitting

Loss curve

Mini-VGGT loss curve

The total, camera, and depth losses decrease during training. The final evaluation yields a total loss of -0.1489, a camera loss of 0.0023, and a depth loss of -0.1601.

Metric Initial Final
Depth AbsRel 0.6186 0.0381
Depth δ1 0.2081 0.9536
Mean rotation error (degrees) 49.73 0.17
Mean translation-direction error (degrees) 90.00 0.37

GT and predicted geometry

Ground truth:

Overfit ground-truth geometry

Predicted camera + depth:

Overfit predicted geometry

3.2. Training and Evaluation

Mini-VGGT Training settings.

Normal-training loss curves

Mini-VGGT training and validation loss curves

The plots show total, camera, and depth losses on the training and validation splits.

Validation and test metrics

Both val and test splits are evaluated using the same best checkpoint.

Split Depth AbsRel ↓ Depth δ1 ↑
Validation 0.1406 0.8336
Test 0.1466 0.8029
Split Metric AUC@5° ↑ AUC@15° ↑ AUC@30° ↑
Validation Rotation 0.1770 0.4963 0.6937
Test Rotation 0.1541 0.4713 0.6942
Validation Translation 0.0110 0.0642 0.1793
Test Translation 0.0066 0.0421 0.1429
Validation Joint pose 0.0020 0.0461 0.1541
Test Joint pose 0.0000 0.0275 0.1327

GT and predicted geometry on four validation scenes

Scene 1: 09c1414f1b_frame_000000

Ground truth:

Ground-truth geometry: scene 1

Predicted camera + depth:

Predicted geometry: scene 1

Scene 2: 21d970d8de_frame_000000

Ground truth:

Ground-truth geometry: scene 2

Predicted camera + depth:

Predicted geometry: scene 2

Scene 3: 280b83fcf3_frame_000000

Ground truth:

Ground-truth geometry: scene 3

Predicted camera + depth:

Predicted geometry: scene 3

Scene 4: 2b1dc6d6a5_frame_000000

Ground truth:

Ground-truth geometry: scene 4

Predicted camera + depth:

Predicted geometry: scene 4

Discussion: errors on held-out scenes

Depth error:

As shown in the screenshots, the predictions capture the overall scene structure, but depth boundaries and fine object details are less accurate.

These artifacts are consistent with depth errors and imperfect alignment across views. The depth error on the test set is slightly higher than on the validation set: AbsRel increases from 0.1406 to 0.1466, while δ1 decreases from 0.8336 to 0.8029.

Rotation, translation, and joint pose errors:

3.3. Bonus: Analysis and Exploration

Question: How does the number of input views during training affect accuracy and runtime?

Experimental settings: Three VGGT models are trained with 2, 4, and 8 input views per sample. The 8-view baseline from Section 3.2 is reused. All runs use batch size 16, sampling range 32, image resolution 224 × 224, seed 825, and are trained for 1000 epochs. The model architecture and the optimizer settings are the same as Section 3.2. The DINOv2 encoder remains frozen.

The best checkpoint for each run is selected by its validation loss (using the corresponding training view count). All three best checkpoints are further evaluated with 8 input views on the same validation split (the same as Section 3.2). Evaluation uses the first 32 prepared frames. Training and validation runtime is measured by summing the per-epoch times in the logs.

Results:

  1. Quantitative results

Runtime includes training and validation over 1,000 epochs. All models are evaluated with 8 views.

Training views Depth AbsRel ↓ Depth δ1 ↑ Runtime (hours)
2 0.1692 0.7726 1.22
4 0.1530 0.8186 1.98
8 0.1406 0.8336 3.46
Training views Metric AUC@5° ↑ AUC@15° ↑ AUC@30° ↑
2 Rotation 0.0906 0.3372 0.5344
4 Rotation 0.2133 0.5259 0.7056
8 Rotation 0.1770 0.4963 0.6937
2 Translation 0.0084 0.0432 0.1111
4 Translation 0.0115 0.0740 0.1853
8 Translation 0.0110 0.0642 0.1793
2 Joint pose 0.0023 0.0270 0.0857
4 Joint pose 0.0054 0.0581 0.1653
8 Joint pose 0.0020 0.0461 0.1541

The model trained with 8 views achieves the best depth accuracy. The 4-view model performs slightly better than the 8-view model on rotation, translation, and joint pose AUCs.

Training time increases with the number of views: 1.22, 1.98, and 3.46 hours for 2, 4, and 8 views, respectively.

  1. Qualitative results

Scene: 09c1414f1b_frame_000000. All predictions use 8 evaluation views.

Ground truth:

Ground-truth geometry

Trained with 2 views:

Geometry predicted by the 2-view training model

Trained with 4 views:

Geometry predicted by the 4-view training model

Trained with 8 views:

Geometry predicted by the 8-view training model

In this scene, increasing the number of training views produces a more recognizable scene structure, with fewer scattered points and less distorted surfaces. This is consistent with the improved depth accuracy. However, even the 8-view model still shows blurred object boundaries and imperfect alignment compared with GT.

Possible explanation: More views require more image processing and cross-view attention, which increases runtime. More training views provide additional geometric information, which may improve depth prediction. However, more views may also make cross-view aggregation harder, so pose accuracy may not necessarily improve.

Limitations: Each setting uses only one seed, so the small pose differences may reflect training variability. Checkpoints are selected using different validation view counts during training, which may not always align with 8 view evaluation.

Conclusion: In these experiments, increasing the number of training views improves depth accuracy, with 8 views achieving the best results. Training with 4 views gives slightly better pose AUCs than 8 views and takes less time. More training views therefore do not improve every metric and come at a higher runtime cost.