16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Unprojection

Unprojection scene 0
Unprojection scene 1

Canonicalization

Camera 0 (red frustum) is at the origin, first camera center = (0, 0, 0), valid-point centroid distance = 1.000.

Canonicalization scene 0
Canonicalization scene 1

check_data.py

SampleValid depthPoint reproj. (px)Ray reproj. (px)Centroid dist.
e050c15a8d_frame_0070800.9781.1e-041.1e-041.0000
5f99900f09_frame_0028200.9901.2e-041.1e-041.0000
47b37eb6f9_frame_0027200.9619.9e-058.9e-051.0000
4c5c60fa76_frame_0066900.7521.1e-041.0e-041.0000
Full output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001089862416847609,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

final PSNR 30.96 dB

Overfit visualization
val_epoch000.png

2.2 Training and Evaluation

Settings: default configs/lvsm.json: 8 views, S = 2 source views, sampling range 32, 224 × 224, learning rate 2e-4, batch size 4, MSE loss, except 300 epochs instead of the default 100.

LVSM training curves

Ablations

Validation

Test

Source viewsTarget raysPSNR (dB)SSIM
1correct15.760.5833
1shuffled11.650.4824
1none12.450.5335
2correct16.970.5915
2shuffled11.250.4616
2none12.400.5306
4correct18.100.6108
4shuffled11.290.4576
4none12.440.5411
Source viewsTarget raysPSNR (dB)SSIM
1correct16.150.5628
1shuffled12.250.4667
1none12.770.5144
2correct17.080.5749
2shuffled12.060.4557
2none12.720.5158
4correct18.230.5808
4shuffled11.570.4327
4none12.710.5108

Visualizations

Training, epoch 0

Training, epoch 295

Training epoch 0
Training epoch 295

Evaluation, S = 1

Evaluation, S = 2

S=1 correct
S=2 correct

Evaluation, S = 4

S = 2, shuffled rays

S=4 correct
S=2 shuffled

S = 2, no rays

S=2 none

Discussion

2.3 Bonus: Loss Function

Question. What happens to PSNR, SSIM and visual sharpness when an L1, SSIM or perceptual term is added to the MSE loss, and is the effect larger than the run-to-run variation?

Setup. Every model uses the Q2.2 configuration and differs only in the loss and, for the repeats, the training seed. Runs: MSE; MSE + 1.0 × L1; MSE + w × (1 − SSIM) with w = 1.0, 0.3, 0.1; MSE + 0.1 × perceptual loss (mean L1 distance between frozen ImageNet VGG16 features at relu1_2, relu2_2, relu3_3, relu4_3). MSE, L1 and SSIM with w = 0.1 were repeated with seeds 825, 1 and 2. Sharpness ratio is mean absolute horizontal + vertical image gradient of the prediction divided by that of the ground truth (smaller = blurrier).

Loss weights (seed 825)

Validation

Test

LossPSNR (dB)SSIMSharpness ratio
MSE (baseline)16.140.56010.379
MSE + 1.0 × L116.280.56340.403
MSE + 1.0 × (1 − SSIM)14.870.57510.127
MSE + 0.3 × (1 − SSIM)15.420.58030.163
MSE + 0.1 × (1 − SSIM)15.550.57780.194
MSE + 0.1 × VGG16 perceptual14.700.56150.321
LossPSNR (dB)SSIMSharpness ratio
MSE (baseline)16.260.54310.345
MSE + 1.0 × L116.370.54510.372
MSE + 1.0 × (1 − SSIM)14.880.55510.119
MSE + 0.3 × (1 − SSIM)15.450.56150.151
MSE + 0.1 × (1 − SSIM)15.610.55940.176
MSE + 0.1 × VGG16 perceptual14.740.54290.290

Three seeds (mean ± std)

Validation

Test

LossPSNR (dB)SSIMSharpness ratio
MSE (baseline)16.13 ± 0.000.5589 ± 0.00120.380 ± 0.003
MSE + 1.0 × L116.28 ± 0.040.5635 ± 0.00010.393 ± 0.009
MSE + 0.1 × (1 − SSIM)15.77 ± 0.190.5822 ± 0.00380.201 ± 0.007
LossPSNR (dB)SSIMSharpness ratio
MSE (baseline)16.22 ± 0.030.5411 ± 0.00180.349 ± 0.004
MSE + 1.0 × L116.36 ± 0.030.5446 ± 0.00080.361 ± 0.010
MSE + 0.1 × (1 − SSIM)15.79 ± 0.170.5635 ± 0.00360.184 ± 0.007
Validation curves

Columns: ground truth, then the six seed-825 models

Loss comparison, validation

Results.

Explanation: L1 is less dominated by a few large errors than MSE, a plausible reason for its small, consistent gain. SSIM compares local mean, variance and covariance, so a smooth prediction with the right brightness layout already scores well. Since the model cannot predict fine texture, the SSIM term pushes it toward smooth outputs, raising SSIM and lowering PSNR and sharpness. A higher SSIM here is therefore not better visual quality. The MSE and L1 predictions show the 8 × 8 patch grid, the patch edges also create image gradients, so their sharpness ratio partly counts block edges instead of real detail. The perceptual loss uses pretrained VGG features to compare textures that this small, undertrained model cannot reproduce, which may explain why there is no gain.

Limitations: One weight and one seed for the perceptual loss (a smaller weight or longer training might behave differently), and all comparisons use 100 epochs and S = 2.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit loss curve

Ground truth

Predicted camera + depth

Ground truth
Predicted camera + depth

3.2 Training and Evaluation

Settings: default configs/vggt.json: 8 views, sampling range 32, 224 × 224, learning rate 6e-4, batch size 16, 1000 epochs, loss = 5 × camera + 1 × depth.

Training loss curve
MetricValidationTest
Depth AbsRel (lower is better)0.1410.160
Depth δ10.8400.797
Rotation AUC@5°0.1850.201
Rotation AUC@15°0.5050.496
Rotation AUC@30°0.6900.697
Translation AUC@5°0.0180.018
Translation AUC@15°0.0900.087
Translation AUC@30°0.2080.198
Joint pose AUC@5°0.0070.005
Joint pose AUC@15°0.0710.056
Joint pose AUC@30°0.1860.171

Validation scene 1: 09c1414f1b

Ground truth

Predicted camera + depth

Ground truth
Predicted camera + depth

Validation scene 2: 21d970d8de

Ground truth

Predicted camera + depth

Ground truth
Predicted camera + depth

Validation scene 3: 280b83fcf3

Ground truth

Predicted camera + depth

Ground truth
Predicted camera + depth

Validation scene 4: 2b1dc6d6a5

Ground truth

Predicted camera + depth

Ground truth
Predicted camera + depth

Discussion

3.3 Bonus: Number of Input Views

Question. The model was trained with 8 views. How do accuracy, runtime and memory change when the number of input views changes at evaluation?

Setup. Same best.pt on the validation split with V ∈ {2, 4, 8, 16, 32} views taken from the first 32 prepared frames. Time is wall-clock per scene including data loading and memory is peak allocated GPU memory.

ViewsAbsRelδ1Rot. AUC@15°Trans. AUC@15°Joint AUC@15°s / scenePeak GB
20.1160.8530.5430.1690.1690.080.70
40.1260.8610.4980.0960.0700.060.75
80.1410.8410.5050.0890.0700.080.84
160.1580.8140.4520.0750.0510.270.98
320.1640.7840.4510.0700.0480.551.29
Number of views

Results. More views at test time did not help this model. Depth AbsRel grows and joint pose AUC@15° drops. δ1 peaks at V = 4. Peak memory rises from 0.70 to 1.29 GB. Time per scene barely changes at first, then rises to 0.55s, since global attention cost grows with the total number of tokens.

Explanation. The model was trained with 8 views only. With 16 or 32 views, the global attention and the camera head get inputs they never saw in training, so the accuracy drops.

Limitations. The comparison across V is not exact, because the scene scale is recomputed from the chosen views and the sampled frames differ. The AUC also covers a different number of camera pairs for each V. With V = 2 there is only one pair per scene. The time includes data loading. This is one checkpoint and one seed, and the model was not retrained. Training with different numbers of views would show whether the drop comes from more views or from the mismatch with training.