Assignment 4: Multi-View Transformers

Shruti Jain (spjain)

1. Multi-View Data Preprocessing

1.2. Depth Unprojection

Scene 036bce3393 after unprojection

Scene 076c822ecc after unprojection

Scene 079a326597 after unprojection

Scene 07f5b601ee after unprojection

Scene 07ff1c45bb after unprojection

1.3. Camera and Depth Canonicalization

Scene 036bce3393 after canonicalization

Scene 076c822ecc after canonicalization

Scene 079a326597 after canonicalization

Scene 07f5b601ee after canonicalization

Scene 07ff1c45bb after canonicalization

1.7. Final Check

Canonicalization. Transform cameras and points into the first sampled camera’s coordinate frame. Divide camera translations, depths and points by the distance from that camera to the valid-point centroid. Camera 0 becomes identity at (0, 0, 0), and the centroid distance becomes 1, as shown in the canonicalization screenshots above.

World-space ray directions. Backproject each pixel vector [u, v, 1] using the inverse intrinsics, rotate it with the camera-to-world rotation, and normalize it to unit length. This gives its direction in the canonical world frame. The camera center is the ray origin; translation affects the origin but not the direction.

Plücker moment. The moment is m = o × d, where o is the ray origin and d its unit direction. It encodes the line’s position relative to the coordinate origin and is unchanged when o moves along that line. The six channels are [dx, dy, dz, mx, my, mz].

check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1. Overfitting

Final PSNR: 36.70 dB.

Columns: source 1, source 2, ground truth, prediction.

LVSM fixed-batch overfit

2.2. Training and Evaluation

Trained for 100 epochs on 224 scenes: 8 views sampled from 32-frame windows, 224 × 224 images, 2 source views, batch size 4, seed 825. The model uses 8 transformer blocks, width 384 and 8 × 8 patches. AdamW: learning rate 0.0002, weight decay 0.05, 200 warmup steps followed by cosine decay; BF16 forward passes with float32 loss. The checkpoint from epoch 94 was selected by validation PSNR.

LVSM training and validation curves

Validation comparison saved during epoch 96 (val_epoch095.png); columns: source 1, source 2, ground truth, prediction.

LVSM validation comparison saved during epoch 96

Evaluation uses all 28 validation scenes, with 8 views from the first 32 prepared frames and seed 825.

Source views Target rays PSNR (dB) SSIM
1 Correct 15.22 0.5568
1 Shuffled 11.80 0.4855
1 None 12.44 0.5290
2 Correct 16.19 0.5589
2 Shuffled 11.45 0.4705
2 None 12.42 0.5266
4 Correct 16.82 0.5739
4 Shuffled 11.58 0.4737
4 None 12.47 0.5372

With correct rays, increasing sources from 1 to 4 changed PSNR by +1.60 dB and SSIM by +0.0171; the highest PSNR used 4 sources. The target-image sets also change with the source count. At 2 sources, shuffling target rays changed PSNR/SSIM by -4.73 dB/-0.0884; removing them changed these by -3.77 dB/-0.0323. The saved validation comparison captures broad colors and layout, but predictions remain blurred and lose table/chair edges.

Evaluation visualizations: all nine settings

Columns: source view(s), ground truth, prediction.

1 source view, correct target rays.

1 source view, correct target rays

1 source view, shuffled target rays.

1 source view, shuffled target rays

1 source view, all-zero target rays.

1 source view, all-zero target rays

2 source views, correct target rays.

2 source views, correct target rays

2 source views, shuffled target rays.

2 source views, shuffled target rays

2 source views, all-zero target rays.

2 source views, all-zero target rays

4 source views, correct target rays.

4 source views, correct target rays

4 source views, shuffled target rays.

4 source views, shuffled target rays

4 source views, all-zero target rays.

4 source views, all-zero target rays

2.3. Bonus: Adding L1 to MSE

MSE and MSE + 0.1 L1 were compared by fine-tuning the same best baseline weights for five additional epochs. Both branches used fresh AdamW optimizers at learning rate 0.00002, batch size 4, two source views, BF16 and seed 825, with identical ordered training scenes and frames in every epoch. The final checkpoints were evaluated on the same 28 validation scenes.

Objective PSNR (dB) SSIM
Shared starting checkpoint 16.1854 0.558877
MSE continuation 16.1591 0.558723
MSE + 0.1 L1 16.1772 0.558515

Controlled LVSM loss comparison

Adding L1 changes PSNR by +0.0181 dB and SSIM by −0.000208 relative to the MSE continuation. Both remain below the starting checkpoint on both metrics. The matched examples remain blurred, with no clear recovery of fine furniture edges. This short comparison shows no clear overall gain; the small loss weight and brief fine-tuning may limit the change. One seed and five epochs do not establish performance after training from scratch.

Columns: source 1, source 2, ground truth, prediction. MSE continuation:

MSE continuation after five epochs

MSE + 0.1 L1 after the same five epochs:

MSE plus L1 after five epochs

3. Multi-View Geometry with Mini-VGGT

3.1. Overfitting

VGGT overfit total, camera and depth losses

VGGT overfit ground-truth and predicted geometry

3.2. Training and Evaluation

Trained for 1,000 epochs (14,000 updates) on 224 scenes using 8 views from 32-frame windows, 224 × 224 images, batch size 16 and seed 825. Frozen DINOv2 ViT-B/14; 8 frame/global attention pairs, width 384, 6 heads; 4 camera-head blocks. AdamW: learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine decay to 5% of the initial rate. Total loss = 5 × camera loss + depth loss. BF16 forward passes with float32 losses; validation every 5 epochs. The best validation-loss checkpoint is epoch 850.

VGGT full-training total, camera and depth losses

Full validation and test evaluation: 8 views from the first 32 prepared frames, 224px, seed 825, batch size 4. Each split contains 28 scenes and 784 valid camera pairs. AUC values range from 0 to 1.

Metric Validation Test
Depth AbsRel ↓ 0.136629 0.163383
Depth δ1 ↑ 0.855322 0.799435
Rotation AUC 5° ↑ 0.197959 0.203316
Rotation AUC 15° ↑ 0.497874 0.520918
Rotation AUC 30° ↑ 0.688988 0.708121
Translation AUC 5° ↑ 0.008163 0.006122
Translation AUC 15° ↑ 0.067432 0.057738
Translation AUC 30° ↑ 0.170068 0.169430
Joint pose AUC 5° ↑ 0.001786 0.002551
Joint pose AUC 15° ↑ 0.050680 0.041752
Joint pose AUC 30° ↑ 0.149915 0.149575

The four held-out scenes retain broad layout, but predicted surfaces are thicker and warped. The tall structure in 280b83fcf3 is compressed and curved; the long pale wall in 21d970d8de bows; camera spacing and surface alignment differ in 2b1dc6d6a5. These views combine camera and depth errors, so they do not isolate either cause. Translation-direction AUC is much lower than rotation AUC, limiting joint pose accuracy. Test depth is worse than validation (AbsRel 0.1634 versus 0.1366).

Ground truth and predicted geometry: 09c1414f1b_frame_000000

Ground truth and predicted geometry: 21d970d8de_frame_000000

Ground truth and predicted geometry: 280b83fcf3_frame_000000

Ground truth and predicted geometry: 2b1dc6d6a5_frame_000000

3.3. Bonus: Number of Input Views

The epoch-850 checkpoint, trained with 8 views, was evaluated with 2, 4, 8 and 16 nested input views on all 28 validation scenes. Each setting used the first 32 prepared frames, 224 × 224 images, seed 825 and batch size 1. Metrics are reported for all input views and for the same first two views to distinguish added context from changes in the evaluated frames.

Views All AbsRel ↓ All δ1 ↑ Common-two AbsRel ↓ Common-two δ1 ↑ Common-pair pose AUC 15° ↑
2 0.1199 0.8552 0.1199 0.8552 0.0119
4 0.1270 0.8622 0.1258 0.8591 0.0619
8 0.1367 0.8554 0.1325 0.8582 0.0762
16 0.1674 0.8005 0.1625 0.8142 0.0667
Views Median forward (ms) Peak allocated (GiB) Peak reserved (GiB)
2 38.87 0.690 0.703
4 34.74 0.708 0.744
8 35.42 0.747 0.783
16 50.04 0.822 0.848

Measured on the L40S with BF16, three warmup forwards per setting and one synchronized forward per scene. Timing excludes data transfer, metrics and file I/O. Allocated memory includes model and inputs; reserved memory includes allocator caching. These are inference measurements.

VGGT accuracy, latency and memory versus number of views

Common-pair pose AUC at 15° improves from 0.0119 with 2 views to 0.0762 with 8, then falls to 0.0667 with 16. Common-view depth AbsRel worsens as views increase; δ1 peaks at 4 views. Sixteen views also use the most memory and take the longest. Additional context may help relative pose, while the change from the 8-view training setting may hurt depth. The loader recomputes scene normalization for each view count, so even the common-view comparison retains a scale confound. All-view metrics additionally use different frame/pair sets. One checkpoint, one seed and one timed pass per scene do not establish a general speed ordering for 2–8 views.

Matched frames 0 and 290 from scene 09c1414f1b, shown in meters with a shared 0–4.70 m range. Broad depth structure is retained, but chair legs, table edges and bookshelf details remain smoothed; added views show no clear detail recovery. White pixels are GT-invalid and masked in both depth panels.

2 input views:

Depth on the same two frames with 2 input views

16 input views:

Depth on the same two frames with 16 input views