16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Unprojection

Unprojected training points and cameras

Canonicalization

Canonicalized geometry, first camera center and valid-point centroid distance

Canonicalization: apply the inverse of camera 0's pose, then divide translations, depth and points by s = ‖mean(p_valid)‖. Camera 0 becomes identity and the valid-point centroid is one unit from the origin.

World-space ray directions: normalize R K⁻¹[u, v, 1]ᵀ, where R is the camera-to-world rotation. The ray origin is the camera center.

Plücker coordinates: [d, m], where d is the unit ray direction and m = o × d is the moment, encoding the ray's offset from the origin.

check_data.py output

{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 9.41436446737498e-05, "centroid_distance": 1.0},
{"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00011108566104667261, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
{"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.591897105565295e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
{"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.0001089862416847609, "max_ray_reprojection_error_px": 0.00010141761595150456, "centroid_distance": 0.9999999403953552}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}

2. Novel-View Synthesis with Mini-LVSM

2.1. Overfitting

Final overfit PSNR: 30.04 dB.

LVSM overfit visualization

2.2. Training and Evaluation

Training settings: 100 epochs; batch size 4; 2 source views; 8 total views at 224×224; sampling range 32; AdamW; learning rate 0.0002; weight decay 0.05; 200 warmup steps; cosine decay; seed 825.

LVSM training curves

Training visualization

LVSM training-time validation visualization

More source views improve reconstruction by providing more observations. Shuffling or zeroing target rays reduces PSNR and SSIM, showing dependence on camera conditioning; shuffled rays are more harmful here.

Evaluation

S=1, correct rays: PSNR 15.19 dB; SSIM 0.5579.

S=1, correct rays

S=1, shuffled rays: PSNR 11.80 dB; SSIM 0.4871.

S=1, shuffled rays

S=1, zeroed rays: PSNR 12.46 dB; SSIM 0.5302.

S=1, zeroed rays

S=2, correct rays: PSNR 16.14 dB; SSIM 0.5602.

S=2, correct rays

S=2, shuffled rays: PSNR 11.50 dB; SSIM 0.4722.

S=2, shuffled rays

S=2, zeroed rays: PSNR 12.44 dB; SSIM 0.5274.

S=2, zeroed rays

S=4, correct rays: PSNR 16.88 dB; SSIM 0.5756.

S=4, correct rays

S=4, shuffled rays: PSNR 11.59 dB; SSIM 0.4732.

S=4, shuffled rays

S=4, zeroed rays: PSNR 12.45 dB; SSIM 0.5372.

S=4, zeroed rays

3. Multi-View Geometry with Mini-VGGT

3.1. Overfitting

VGGT overfit total, camera and depth loss curves

Ground truth

Overfit ground-truth geometry

Prediction

Overfit predicted geometry

3.2. Training and Evaluation

Training settings: 1,000 epochs; batch size 16; 8 views at 224×224; sampling range 32; AdamW; learning rate 0.0006; weight decay 0.01; 10 warmup epochs; cosine decay; camera/depth loss weights 5/1; seed 825.

VGGT training and validation total, camera and depth loss curves

Validation: depth AbsRel 0.1443; depth δ1 81.58%.

Validation rotation AUC — 5°: 0.1893; 15°: 0.5167; 30°: 0.6939.

Validation translation AUC — 5°: 0.0133; 15°: 0.0912; 30°: 0.2019.

Validation joint pose AUC — 5°: 0.0038; 15°: 0.0753; 30°: 0.1843.

Test: depth AbsRel 0.1557; depth δ1 79.14%.

Test rotation AUC — 5°: 0.2434; 15°: 0.5707; 30°: 0.7584.

Test translation AUC — 5°: 0.0168; 15°: 0.0720; 30°: 0.1770.

Test joint pose AUC — 5°: 0.0061; 15°: 0.0575; 30°: 0.1679.

On held out scenes, depth recovers broad room surfaces but loses fine structure. Camera errors, especially translation, misalign points across views and reduce joint-pose AUC on held-out scenes.

09c1414f1b

Ground truth

09c1414f1b ground-truth geometry

Prediction

09c1414f1b predicted geometry

21d970d8de

Ground truth

21d970d8de ground-truth geometry

Prediction

21d970d8de predicted geometry

280b83fcf3

Ground truth

280b83fcf3 ground-truth geometry

Prediction

280b83fcf3 predicted geometry

2b1dc6d6a5

Ground truth

2b1dc6d6a5 ground-truth geometry

Prediction

2b1dc6d6a5 predicted geometry