16-825 Assignment 4: Multi-View Transformers

Andrew ID: robertre

1. Multi-View Data Preprocessing

1.1-1.2 Unprojection

Unprojected points and cameras of the 8 sampled views in the original world frame (meters). Scene 036bce3393_frame_004470.

1.3 Canonicalization

The same views after canonicalization. First camera center: (0.0000, -0.0000, 0.0000). Valid-point centroid distance: 1.000 normalized units (scene scale 1.867 m).

1.7 Final Check

$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011279189493507147, "max_ray_reprojection_error_px": 0.00010904046939685941, "centroid_distance": 1.0},
    {"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
    {"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
    {"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00010924567322945222, "max_ray_reprojection_error_px": 0.00010257320536766201, "centroid_distance": 1.0}
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Explanations

Canonicalization: Canonicalization changes the reference frame so that every camera pose and point is expressed in the frame of the first sampled camera of the scene. With this operation, the first camera has identity rotation and zero translation. By canonicalizing the cameras, we also normalize the scale by dividing translation, depth, and points by the distance from camera 0 to the centroid of all valid points. Canonicalization is necessary to remove the arbitrary world frame and scale of each scene so the training use samples with consistent conventions.

World-space ray directions: To obtain world space rays, we first back-project a given pixel to camera space, and rotate it to world space using the camera to world transformation. We then normalize it by scaling it to unit length. The advantage of using world space rays is that all the rays from different cameras share the same reference frame which is the first sampled camera's canonical frame.

Plücker moment: A ray has a camera center and a unit direction (o, d). The plucker coordinates describe the ray with 6 numbers (d, o × d). The cross product is used instead of just the camera center because the cross product does not depend on which point we pick as an origin. Mathematically, (o + t*d) × d = o × d. Therefore, using just (o, d) could have ambiguity, while (d, o × d) is unique for each ray.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Overfit on one fixed batch for 2000 steps. Final PSNR: 30.93 dB (SSIM 0.870).

Rows: target views. Columns: source 1, source 2, ground truth, prediction.

2.2 Training and Evaluation

Training settings

Modelpatch size 8, width 384, 8 transformer blocks, head dim 64, MLP ratio 4, QK-norm (14.6M parameters)
Views8 per sample (2 source, 6 target), from a 32-frame window, 224 x 224
Batch size4 samples (24 target images)
LossMSE
OptimizerAdamW, learning rate 2e-4, weight decay 0.05, betas (0.9, 0.95), gradient clipping 1.0, bf16 autocast
Schedule200 warmup steps, then cosine decay to 10%
Training length100 epochs (5,600 steps), 53 min on one A100 40 GB
Checkpointbest validation PSNR (epoch 99)
Evaluationvalidation split, 8 views from the first 32 prepared frames, seed 825

Training curves

Validation metrics (PSNR / SSIM)

Source viewsCorrect raysShuffled raysNo rays
115.19 / 0.55811.80 / 0.48712.48 / 0.524
216.12 / 0.56011.49 / 0.47212.44 / 0.525
416.87 / 0.57511.58 / 0.47212.46 / 0.536

Training visualizations

Validation batch during training. Left: epoch 0. Right: epoch 95. Columns: source views, ground truth, prediction.

Evaluation visualizations

Correct raysShuffled raysNo rays
S=1
S=2
S=4

Columns in each image: source views, ground truth, prediction.

Discussion

Number of source views: Having more input views improve the novel view synthesis. However, the gain will get smaller as more views are added. Since attention can accept any number of source view tokens, the model can take in more than 2 views even though it was only trained on 2 views. More source views give the model more information to infer the global geometry and appearance, and therefore help the model generalize better to novel views.

Ray-conditioning ablation: The model is clearly using the target ray information. Based on the ablation, the model performs a lot better when the correct target view ray information is given. At 2 context views, correct rays give 16.12 dB, no rays 12.44 and shuffled rays 11.49. Shuffled rays confuse the model even more and perform less than no rays.

2.3 Bonus: SSIM Loss

Question: How does adding an SSIM term to the MSE loss change PSNR, SSIM and visual sharpness?

Settings: Same as 2.2, but the training loss is MSE + 0.2 · (1 - SSIM). The validation loss is still MSE.

Source viewsMSE (2.2)MSE + SSIM
PSNRSSIMPSNRSSIM
115.190.55814.740.575
216.120.56015.370.579
416.870.57515.740.592

Correct target rays, validation split.

S=2. Left: MSE. Right: MSE + SSIM.

Results and conclusions: Including the SSIM term in the loss gives a tradeoff. It improved the structural similarity of the image but lowered the PSNR because PSNR scores the per pixel error directly and was downweighted due to the SSIM term. Visually, the predicted images with the SSIM term are much smoother, and the blocky patterns from the model without the ssim loss mostly disappeared. This test used only one ssim weight of 0.2 so the tradeoff might be different with other weights

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

One fixed training clip (8 views), 40 epochs of 25 steps, learning rate 1e-3. Final: total loss -0.159 (camera 0.002, depth -0.169), depth AbsRel 0.038, δ1 0.954, pose AUC@30 0.998.

Top: ground truth (left) and predicted cameras and depth (right), seen from behind the cameras and from above. Bottom: RGB, ground-truth depth and predicted depth for all 8 views.

3.2 Training and Evaluation

Training settings

Modelfrozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (16 blocks), width 384, 6 heads, camera head with 4 cross-view blocks, depth head with RGB refinement (57.7M trainable parameters)
Views8 per sample, from a 32-frame window, 224 x 224
Batch size16
Loss5 x camera loss + 1 x depth loss
OptimizerAdamW, learning rate 6e-4, weight decay 0.01, gradient clipping 1.0, bf16 autocast
Schedule10 warmup epochs, then cosine decay to 5%
Training length1000 epochs (14,000 steps), 3.3 h on one A100 40 GB
Checkpointbest validation loss (epoch 843)
Evaluation8 views from the first 32 prepared frames, 224 px, seed 825, 28 scenes per split (784 view pairs)

Loss curves

Metrics

SplitAbsRelδ1Rotation AUCTranslation AUCPose AUC
5°15°30°5°15°30°5°15°30°
Validation0.1320.8490.1960.5380.7150.0180.0780.1910.0050.0560.171
Test0.1560.8090.1790.5110.7180.0140.0740.1770.0020.0520.157

Ground truth vs. prediction (validation scenes)

For each scene: ground truth (left) and predicted cameras and depth (right), seen from behind the cameras and from above, then RGB, ground-truth depth and predicted depth for all 8 views.

Discussion

On held-out scenes the model recovers the overall room layout, but test is slightly worse than validation. Translation is much worse than rotation since the cameras are very close together, so small position errors cause large direction errors. The predicted depth is smooth and blocky so thin structures like chair legs merge into blobs.

3.3 Bonus: Number of Input Views

Question: How does the number of input views affect accuracy, runtime and memory?

Settings: The 3.2 model (trained with 8 views), evaluated on the validation split with 2, 4, 8, 16 and 32 views from the first 32 prepared frames. Batch size 1 on one A100 40 GB. Time is the model forward pass per sample, after one warm-up pass. Memory is the peak allocated GPU memory, including the model weights.

ViewsAbsRelδ1Rotation AUC@30Translation AUC@30Pose AUC@30Time (ms)Memory (GB)
20.1080.8750.6760.2130.183360.69
40.1290.8570.6830.2020.174370.71
80.1320.8490.7150.1910.172390.75
160.1540.8140.6770.1700.145430.83
320.1600.7930.6810.1640.145500.97

Results and conclusions: As input views increase, the runtime and memory increase as expected because the attention needs to handle more tokens. More views don't always mean better accuracies and the rotation AUC@30 starts to drop after 8 views. The translation and pose AUC drops as view counts increase as well. AbsRel rises from 0.108 with 2 views to 0.160 with 32 views. Therefore, more views could introduce challenge for the model for accurate geometry prediction. The limitation of this evaluation is that the model was only trained on 8 views, and that the evaluation for each view count use different frames. Therefore, this comparison is not perfectly apples-to-apples.