16-825 Assignment 4: Multi-View Transformers

Graham Felton | gtf

1. Multi-View Data Preprocessing

Unprojection (Q1.1–Q1.2)

Preview of unprojected 3d points

Canonicalization (Q1.3)

Preview of unprojected 3d points

Final check (Q1.7)

scripts/check_data.py --split train --max-samples 4 on 224 training samples:

SampleValid depth fractionMax point reprojection error (px)Max ray reprojection error (px)Centroid distance
e050c15a8d_frame_0070800.9781.11e-049.65e-051.0000
5f99900f09_frame_0028200.9901.11e-041.08e-041.0000
47b37eb6f9_frame_0027200.9619.59e-058.22e-051.0000
4c5c60fa76_frame_0066900.7521.08e-049.54e-051.0000
Batch shapes
FieldShape
rgb(2, 8, 3, 224, 224)
depth(2, 8, 1, 224, 224)
depth_valid(2, 8, 1, 224, 224)
K(2, 8, 3, 3)
frame_numbers(2, 8)
T_world_from_camera(2, 8, 4, 4)
original_world_from_reference(2, 4, 4)
scene_scale(2,)
points_world(2, 8, 3, 224, 224)
ray_origins(2, 8, 3, 224, 224)
ray_directions(2, 8, 3, 224, 224)
plucker(2, 8, 6, 224, 224)

Explanations

Canonicalization gives every sample the same conventions: all poses are expressed in camera 0's frame, and translations, depth and points are divided by the distance from the origin to the valid-point centroid, so camera 0 is the identity and the centroid lies at distance 1. A pixel's world-space ray direction is K-1[u, v, 1]T rotated by the camera-to-world rotation R and normalized to unit length, and its origin is the camera center. The Plücker moment m = o × d is perpendicular to the plane that contains the ray and the world origin and does not depend on which point of the ray is used, so [d, m] describes the line itself.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

One fixed training batch, 2000 steps: final PSNR 30.98 dB, SSIM 0.871 (6 min on NVIDIA A100-SXM4-40GB).

Overfit batch. Columns: source views, ground truth, prediction.
Overfit batch. Columns: source views, ground truth, prediction.

2.2 Training and evaluation

Model
patch 8, width 384, 8 layers, head dim 64, MLP ratio 4.0, QK-norm True (14.6M parameters)
Data
8 views per sample, 2 source + 6 target, sampling range 32, 224px, batch size 4
Optimization
AdamW, lr 0.0002, betas (0.9, 0.95), weight decay 0.05, 200 warmup steps then cosine decay, grad clip 1.0, bf16 autocast
Schedule
100 epochs × 56 steps
Hardware
1× NVIDIA A100-SXM4-40GB, 1.09 h, peak memory 9.7 GB
Training curves. Final validation PSNR 16.13 dB, SSIM 0.560.
Training curves. Final validation PSNR 16.13 dB, SSIM 0.560.

Validation visualizations during training (columns: source views, ground truth, prediction):

Validation, epoch 0
Validation, epoch 0
Validation, epoch 50
Validation, epoch 50
Validation, epoch 95
Validation, epoch 95

Ablations: source views and target-ray conditioning (validation split)

Source viewsCorrect raysShuffled raysNo raysTarget images
PSNRSSIMPSNRSSIMPSNRSSIM
115.190.55811.810.48712.460.527196
216.130.56011.500.47212.440.526168
416.890.57511.590.47312.460.537112
S = 1, correct target rays
S = 1, correct target rays
S = 1, shuffled target rays
S = 1, shuffled target rays
S = 1, no target rays (all zeros)
S = 1, no target rays (all zeros)
S = 2, correct target rays
S = 2, correct target rays
S = 2, shuffled target rays
S = 2, shuffled target rays
S = 2, no target rays (all zeros)
S = 2, no target rays (all zeros)
S = 4, correct target rays
S = 4, correct target rays
S = 4, shuffled target rays
S = 4, shuffled target rays
S = 4, no target rays (all zeros)
S = 4, no target rays (all zeros)

With the correct target rays, the reconstruction quality increases as more source views are added. Once we begin shuffling or excluding rays, the image reconstruction quality decreases, as evidenced by lower PSNR and SSIM values.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

One training clip, 40 epochs × 25 steps (5 min): depth δ1 0.955, AbsRel 0.037, rotation error 0.25°, translation error 0.54°, pose AUC@30 0.993.

Overfit loss curves: total, camera and depth.
Overfit loss curves: total, camera and depth.
Overfit clip: ground truth and predicted geometry
Overfit clip: GT (left) vs. predicted (right) geometry.

3.2 Training and evaluation

Model
frozen dinov2_vitb14, 8 frame/global pairs, width 384, 6 heads (57.7M trainable parameters)
Data
8 views, sampling range 32, 224px, batch size 16
Optimization
AdamW, lr 0.0006, weight decay 0.01, 10 warmup epochs then cosine decay to 0.05× lr, loss weights camera 5.0 / depth 1.0, bf16
Schedule
1000 epochs × 14 steps; best.pt from epoch 721 (lowest validation loss)
Hardware
1× NVIDIA A100-SXM4-40GB, 8.15 h, peak memory 14.7 GB
Training loss curves: total, camera and depth.
Training loss curves: total, camera and depth.

Metrics (best.pt; 8 views from the first 32 prepared frames, 224px, seed 825)

MetricValidation (28 scenes)Test (28 scenes)
Depth AbsRel ↓0.14260.1613
Depth δ1 ↑0.82850.7915
Rotation AUC@5° ↑0.17650.2010
Rotation AUC@15° ↑0.48270.5091
Rotation AUC@30° ↑0.69090.7088
Translation AUC@5° ↑0.00970.0092
Translation AUC@15° ↑0.06170.0511
Translation AUC@30° ↑0.17050.1418
Joint pose AUC@5° ↑0.00380.0028
Joint pose AUC@15° ↑0.04970.0400
Joint pose AUC@30° ↑0.15610.1244
Mean rotation error (°)12.2610.03
Mean translation-direction error (°)43.7941.88

GT vs. predicted geometry on four scenes

Validation scene 1: ground truth and predicted geometry
Validation scene 1: GT (left) vs. predicted (right) geometry.
Validation scene 2: ground truth and predicted geometry
Validation scene 2: GT (left) vs. predicted (right) geometry.
Validation scene 3: ground truth and predicted geometry
Validation scene 3: GT (left) vs. predicted (right) geometry.
Validation scene 4: ground truth and predicted geometry
Validation scene 4: GT (left) vs. predicted (right) geometry.

Validation and test results are close so the model generalizes reasonably to unseen scenes. Translation is the main failure, as AUC@30 is only 0.171 and 0.142.