16-825 · Learning for 3D Vision

Multi-View
Transformers

From camera-aware data preparation to novel-view synthesis and feed-forward 3D reconstruction.

Leo XuAssignment 4Fall 2026
RGB
Rays
Tokens
3D
Σ
01 — 03 Complete report
Overview

One geometry contract,
two transformer tasks.

The project first canonicalizes multi-view ScanNet++ clips, then reuses that representation for two complementary problems. Mini-LVSM renders an unseen target view from RGB source views and Plücker rays. Mini-VGGT predicts camera parameters and dense depth directly from multiple images.

224training scenes
8views per clip
224²input resolution
3/3parts completed
01

Multi-view data preprocessing

Make every scene speak the same coordinate language.

Geometry pipeline

After a provided center crop, I resize intrinsics using the same half-pixel mapping as align_corners=False: c′ = (c + 0.5)s − 0.5. Optical-axis depth is unprojected as Xcam = zK−1[u,v,1]T, then transformed to world coordinates by the camera-to-world pose.

For canonicalization, every pose is left-multiplied by the inverse first-camera pose. The shared scale is the distance from the first camera to the centroid of all valid points. Camera translations, depth, and points are divided by this one value—without subtracting the centroid—so camera 0 stays at identity and the centroid distance becomes one.

Before

World-frame unprojection

Unprojected world-space points and cameras
After

Canonical first-camera frame

Canonicalized points and cameras with the first camera at the origin and centroid distance one

Final data check

Four samples passed every invariant.

1.0000centroid distance
1.23×10⁻⁴ pxmax point reprojection
1.11×10⁻⁴ pxmax ray reprojection

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
All four samples passed. Both reprojection errors are roughly two orders of magnitude below the required 0.01-pixel threshold. Shapes, finite values, rotations, ray norms, depth consistency, and Plücker orthogonality also passed.

02

Novel-view synthesis · Mini-LVSM

Ask a camera ray where the pixels should be.

Source RGB + rays→8×8 patches→Full-attention transformer→Target RGB

Architecture & training

Source tokens concatenate RGB shifted to [−1,1] with six-channel Plücker rays. Target queries contain only target rays, so the model never sees target RGB. A shared 2D sine/cosine embedding marks patch location; all source tokens and one target’s query tokens form a decoder-only sequence. The last query tokens decode to RGB patches through a sigmoid and are supervised with mean squared error.

Sources / targets
2 / 6
Width / layers
384 / 8
Patch size
8
Optimizer
AdamW · 2×10⁻⁴
Training
100 epochs · bf16

Q2.1 · fixed-batch overfit

Memorization check: 30.98 dB

The fixed batch reached 30.98 dB PSNR, 0.8715 SSIM, and 0.00085 MSE after 2,000 steps, confirming correct token ordering, decoding, and supervision.

LVSM overfit source, ground truth, and prediction comparison

Q2.2 · full training

Quality improves steadily.

The best checkpoint occurred at epoch 99 with 16.12 dB PSNR and 0.5601 SSIM. The gap between train and validation curves remained small.

LVSM training loss and PSNR curves

Target-ray ablation

Pose conditioning matters more than another source view.

PSNR and SSIM across LVSM ablations
SourcesConditioningPSNR ↑SSIM ↑
1Correct15.190.5583
1Shuffled11.800.4874
1None12.470.5285
2Correct16.120.5601
2Shuffled11.490.4721
2None12.440.5270
4Correct16.860.5748
4Shuffled11.590.4729
4None12.460.5368

With correct rays, adding sources improves PSNR from 15.19 to 16.86 dB. Shuffling target rays costs 3.4–5.3 dB; removing rays also collapses performance near 12.45 dB. The model therefore uses explicit camera conditioning rather than merely copying appearance from the sources.

Q2.2 · training visualization

Held-out novel-view synthesis.

Source views provide appearance while the target Plücker rays specify the requested view. The prediction recovers the room layout and dominant colors, with remaining blur around fine geometry and occlusion boundaries.

LVSM full-training source views, ground truth, and prediction

Q2.3 · bonus: model capacity

7.3× fewer parameters for only 0.41 dB.

14.6Mbaseline · 16.12 dB
2.0Mcompact · 15.71 dB
2.44×faster training
−0.007SSIM difference

The controlled compact model halves width (192) and depth (4 layers) while keeping data, optimization, patch size, and 100 epochs fixed. It trains in 19.4 minutes versus 47.4 minutes. The modest quality loss suggests the baseline is over-capacity for this data budget, though its advantage grows from 0.28 dB at one source to 0.60 dB at four.

03

Multi-view geometry · Mini-VGGT

Recover cameras and depth in one forward pass.

Alternating attention

A frozen DINOv2 ViT-B/14 encodes RGB patches. Eight pairs of geometry blocks alternate between attention within each frame and global attention across every view. Concatenating the final frame and global features preserves local image evidence while exposing cross-view correspondences.

The camera head predicts world-to-camera translation, quaternion, and horizontal/vertical field of view. The dense head reconstructs positive depth with an exponential activation, adds an RGB-guided residual, and predicts confidence ≥ 1.

Views
8
Width / heads
384 / 6
Attention pairs
8 (16 blocks)
Camera head
4 blocks
Training
1,000 epochs · bf16

Q3.1 · single-scene overfit

Geometry converges to the fixed clip.

Loss fell from 5.48 to −0.177. Depth AbsRel improved from 0.619 to 0.035, δ1 rose from 0.208 to 0.957, and rotation / translation errors dropped to 0.25° / 0.46°. A negative final objective is valid because the confidence term includes −0.2 log(confidence).

VGGT overfit loss curve
Ground-truth and predicted geometry for the Mini-VGGT overfit clip

Ground truth and predicted geometry after single-scene overfitting.

Q3.2 · full training

Best checkpoint at epoch 617.

The model trained on 224 scenes for 1,000 epochs. The best checkpoint was selected by validation loss (0.8321), not by the final epoch. Curves show noisy scene-to-scene supervision but a stable long-run decline.

VGGT full training loss curve
Ground-truth and predicted Mini-VGGT geometry for four held-out validation scenes

Ground truth and predicted geometry for four different held-out validation scenes.

Validation · 28 scenes

0.1473Depth AbsRel ↓
δ10.8223
Rotation AUC @ 30°0.6851
Translation AUC @ 30°0.1804
Joint AUC @ 30°0.1605

Test · 28 scenes

0.1633Depth AbsRel ↓
δ10.7966
Rotation AUC @ 30°0.7198
Translation AUC @ 30°0.1553
Joint AUC @ 30°0.1369
All required pose AUC metrics
SplitMetric@ 5°@ 15°@ 30°
ValRotation0.17370.49430.6851
ValTranslation0.01280.07210.1804
ValJoint0.00380.05490.1605
TestRotation0.19520.51830.7198
TestTranslation0.00710.05260.1553
TestJoint0.00130.03330.1369

Depth and relative rotation generalize substantially better than translation direction. Translation is especially sensitive to small-baseline clips and scene ambiguity; because joint pose uses the worse of rotation and translation error, translation dominates the joint metric. Test depth is slightly worse than validation, while rotation AUC is slightly better, indicating that error modes vary by scene composition rather than a uniform generalization gap.

Q3.3 · bonus: number of views

More views are not automatically easier.

ViewsAbsRel ↓δ1 ↑Rot AUC 30 ↑Trans AUC 30 ↑Joint AUC 30 ↑Runtime
20.11440.85710.68570.25120.191710.71 s
40.12970.86010.65580.20200.160512.74 s
80.14730.82230.68510.18040.160512.28 s

The same eight-view-trained checkpoint was evaluated without fine-tuning. Two views unexpectedly gave the best AbsRel and joint pose AUC, while four gave the best δ1. Higher view counts also evaluate more pixels and many more pose pairs (28 → 168 → 784), often spanning harder baselines. Runtime includes loading and reporting overhead, so it does not isolate the expected quadratic global-attention scaling. A stronger follow-up would train separate models or randomize view count during training.

Takeaways

Coordinates are the quiet architecture.

01

Canonicalization stabilizes learning.

A first-camera frame and shared scene scale remove arbitrary global pose and units while preserving multi-view geometry.

02

Rays make rendering controllable.

LVSM’s sharp ablation drop shows that explicit world-space ray queries—not source appearance alone—select the rendered view.

03

Depth is easier than translation.

VGGT learns useful depth and rotation, while translation-direction ambiguity remains the dominant held-out pose error.

Back to top ↑
Expanded result