Assignment 4: Multi-View Transformers

Sumanth Yegurla · syegurla · 16-825, Fall 2026

1. Multi-View Data Preprocessing

RGB, depth, and camera parameters are cropped and resized together to 224 × 224. The implemented geometry places points, cameras, and rays in a consistent coordinate frame.

1.2 Unprojection

Each depth pixel becomes a 3D point through z K⁻¹[u, v, 1], followed by its camera-to-world transform. The viewer below shows the resulting colored point cloud and camera frustums in the original world frame.

1.3 Canonicalization

The first sampled camera defines the reference frame. Camera translations, depth, and points are scaled so the valid-point centroid is one unit from the origin. Camera 0 is at (0, 0, 0); the centroid is not translated to the origin.

1.1, 1.4–1.6 Implementation notes

Intrinsics. Resizing uses the pixel-center mapping u′ = sx(u + 0.5) − 0.5 and the corresponding vertical transform after cropping.

Rays. A ray starts at the camera center and points along normalized R K⁻¹[u, v, 1]. Its Plücker representation is [d, o × d]; the moment does not change when the origin moves along that ray.

Sampling. Training randomly samples views within a temporal window. Validation and test use deterministic prepared samples and the provided evaluation geometry.

1.7 Final geometry check

SequenceViewsValid depthPoint error (px)Ray error (px)Centroid distance
e050c15a8d / 007080897.8%0.00011060.00009591.000000
5f99900f09 / 002820899.0%0.00012210.00010791.000000
47b37eb6f9 / 002720896.1%0.00009540.00008641.000000
4c5c60fa76 / 006690875.2%0.00011410.00009421.000000

All four checked clips have reprojection errors below 0.00013 pixels. Full numerical results.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

After 2,000 steps on a fixed training batch, PSNR reaches 30.90 dB and SSIM 0.870.

Fixed-batch overfit. Each row: source images, ground-truth target, predicted target.
Fixed-batch overfit. Each row: source images, ground-truth target, predicted target.

2.2 Training and evaluation

Settings: 8 transformer layers, width 384, 8 × 8 patches, 2 source views during training, batch size 4, 100 epochs. AdamW with learning rate 2 × 10⁻⁴, 200 warmup steps, cosine decay, and weight decay 0.05.

Training and validation curves over 100 epochs. The saved plot uses zero-based epoch indices.
Training and validation curves over 100 epochs. The saved plot uses zero-based epoch indices.
First completed epoch
First completed epoch
Epoch 96
Epoch 96

Source views and target-ray conditioning

Source viewsTarget raysPSNR (dB) ↑SSIM ↑
1correct15.1190.5557
1shuffled11.7670.4853
1none12.4950.5318
2correct16.1090.5590
2shuffled11.4450.4709
2none12.4630.5292
4correct16.8320.5734
4shuffled11.5690.4742
4none12.4820.5388

Each image below shows source views on the left, followed by the ground-truth target and prediction. Click an image for its full resolution.

1 source view

Correct rays
Correct rays
Shuffled rays
Shuffled rays
No rays
No rays

2 source views

Correct rays
Correct rays
Shuffled rays
Shuffled rays
No rays
No rays

4 source views

Correct rays
Correct rays
Shuffled rays
Shuffled rays
No rays
No rays

Correct ray conditioning gives substantially higher PSNR than shuffled or absent rays. Additional source views improve the measured score, although the target-image counts differ across settings (196, 168, and 112), so the comparison does not hold the target population fixed.

The held-out predictions recover coarse color and lighting, but remain blurry. Fine furniture edges and scene detail are not reconstructed well, despite the strong fixed-batch overfit result.

2.3 Optional experiment: MSE + L1 loss

Question: does adding absolute-error supervision improve novel-view synthesis? The variant uses L = MSE + 0.1 L1. Architecture, initialization seed, optimizer, sampling, and 100-epoch budget are held fixed. Both checkpoints are selected by validation PSNR and evaluated using the same PSNR/SSIM definitions.

Source viewsMSE PSNRMSE + L1 PSNRΔ PSNRMSE SSIMMSE + L1 SSIM
115.11915.174+0.0550.55570.5574
216.10916.216+0.1060.55900.5611
416.83216.937+0.1050.57340.5760
MSE baseline · 4 source views
MSE baseline · 4 source views
MSE + 0.1 L1 · 4 source views
MSE + 0.1 L1 · 4 source views

The L1 variant yields small gains with correct rays: about 0.11 dB for both two and four source views. The images still show substantial blur; this single-seed result does not establish a large perceptual improvement.

All L1-variant conditioning results

Source viewsTarget raysPSNR (dB) ↑SSIM ↑
1correct15.1740.5574
1shuffled11.7300.4844
1none12.4960.5320
2correct16.2160.5611
2shuffled11.3850.4693
2none12.4710.5286
4correct16.9370.5760
4shuffled11.4810.4711
4none12.5060.5372
1 source view · correct rays
1 source view · correct rays
1 source view · shuffled rays
1 source view · shuffled rays
1 source view · no rays
1 source view · no rays
2 source views · correct rays
2 source views · correct rays
2 source views · shuffled rays
2 source views · shuffled rays
2 source views · no rays
2 source views · no rays
4 source views · correct rays
4 source views · correct rays
4 source views · shuffled rays
4 source views · shuffled rays
4 source views · no rays
4 source views · no rays

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

One fixed training scene, 40 epochs, and 1,000 optimizer steps. Final depth AbsRel is 0.0363, depth δ₁ is 95.6%, and rotation error is 0.24°.

Total, camera, and depth losses during fixed-scene overfitting.
Total, camera, and depth losses during fixed-scene overfitting.
Ground-truth and predicted geometry after overfitting; identical axes and depth colors.
Ground-truth and predicted geometry after overfitting; identical axes and depth colors.

3.2 Training and evaluation

Settings: frozen DINOv2 ViT-B/14 encoder; 8 frame/global attention pairs, width 384, 6 heads; 8 views, sampling window 32, batch size 16. Trained for 1,000 epochs with AdamW, learning rate 6 × 10⁻⁴, 10-epoch warmup, cosine decay, and weight decay 0.01. Camera and depth loss weights are 5 and 1.

Training completed all 14,000 optimizer steps. The checkpoint at epoch 849 achieved the lowest validation loss (0.8583) and is used for every final evaluation below.

Full 1,000-epoch training and validation curves.
Full 1,000-epoch training and validation curves.

Held-out results

MetricValidationTest
Depth AbsRel ↓0.14550.1542
Depth δ₁ ↑0.83000.7895
Rotation error (°) ↓13.84039.4372
Translation-direction error (°) ↓42.655142.7558
Rotation AUC @5° ↑0.16400.2051
Rotation AUC @15° ↑0.46240.5279
Rotation AUC @30° ↑0.66270.7238
Translation AUC @5° ↑0.01430.0105
Translation AUC @15° ↑0.07350.0543
Translation AUC @30° ↑0.17780.1615
Joint pose AUC @5° ↑0.00280.0036
Joint pose AUC @15° ↑0.04930.0412
Joint pose AUC @30° ↑0.14790.1493

28 validation scenes and 28 test scenes. Eight views sampled from the first 32 prepared frames at 224 × 224, seed 825. AUC and δ₁ values are fractions; higher is better.

Depth δ₁ reaches 83.0% on validation and 79.0% on test. Test depth error is slightly higher, while test rotation error is lower. Translation direction remains the weaker pose component: mean errors exceed 42° on both splits, and joint pose AUC is much lower than rotation AUC.

Four held-out scenes

Ground truth and predictions use the same coordinate frame, axis limits, and depth color scale. These are final best-checkpoint results.

Validation scene 1: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 1: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 2: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 2: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 3: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 3: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 4: ground truth and predicted geometry, with RGB and depth comparison.
Validation scene 4: ground truth and predicted geometry, with RGB and depth comparison.

3.3 Optional experiment: Number of input views

Question: how do additional input views affect geometry accuracy and inference cost? The same eight-view-trained checkpoint is evaluated with nested subsets of 2, 4, and 8 images. Scene scale stays fixed. Depth is compared on the same reference-view pixels, and pose on the same camera pair (0, 1).

Input viewsReference AbsRel ↓Reference δ₁ ↑Rotation error (°) ↓Translation error (°) ↓Forward ms/scene ↓Peak GPU GiB ↓
20.191768.3%15.5937.443.541.07
40.154979.4%13.8433.894.551.36
80.128787.2%13.1735.018.471.96

All settings use 28 scenes and the same 1,281,355 valid reference-view pixels. Forward times are batched throughput per scene, with CUDA synchronization and warm-up excluded; data loading and metric calculation are excluded. Peak allocated GPU memory includes the model and shared input batch.

2 input views: the same reference RGB, ground-truth depth, and predicted depth.
2 input views: the same reference RGB, ground-truth depth, and predicted depth.
4 input views: the same reference RGB, ground-truth depth, and predicted depth.
4 input views: the same reference RGB, ground-truth depth, and predicted depth.
8 input views: the same reference RGB, ground-truth depth, and predicted depth.
8 input views: the same reference RGB, ground-truth depth, and predicted depth.

Reference-view depth accuracy increases from 68.3% to 79.4% to 87.2% as views increase from 2 to 4 to 8. Eight views cost about 2.4× the forward time and 1.8× the peak memory of two views. Rotation error improves modestly; translation error is lowest with four views, so the pose benefit is not uniform.

This measures inference-time sensitivity of a model trained with eight views. It does not compare models trained separately at each view count.