World-frame unprojection

16-825 · Learning for 3D Vision
From camera-aware data preparation to novel-view synthesis and feed-forward 3D reconstruction.
The project first canonicalizes multi-view ScanNet++ clips, then reuses that representation for two complementary problems. Mini-LVSM renders an unseen target view from RGB source views and Plücker rays. Mini-VGGT predicts camera parameters and dense depth directly from multiple images.
Multi-view data preprocessing
After a provided center crop, I resize intrinsics using the same half-pixel mapping as align_corners=False: c′ = (c + 0.5)s − 0.5. Optical-axis depth is unprojected as Xcam = zK−1[u,v,1]T, then transformed to world coordinates by the camera-to-world pose.
For canonicalization, every pose is left-multiplied by the inverse first-camera pose. The shared scale is the distance from the first camera to the centroid of all valid points. Camera translations, depth, and points are divided by this one value—without subtracting the centroid—so camera 0 stays at identity and the centroid distance becomes one.


Final data check
python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
All four samples passed. Both reprojection errors are roughly two orders of magnitude below the required 0.01-pixel threshold. Shapes, finite values, rotations, ray norms, depth consistency, and Plücker orthogonality also passed.
Novel-view synthesis · Mini-LVSM
Source tokens concatenate RGB shifted to [−1,1] with six-channel Plücker rays. Target queries contain only target rays, so the model never sees target RGB. A shared 2D sine/cosine embedding marks patch location; all source tokens and one target’s query tokens form a decoder-only sequence. The last query tokens decode to RGB patches through a sigmoid and are supervised with mean squared error.
Q2.1 · fixed-batch overfit
The fixed batch reached 30.98 dB PSNR, 0.8715 SSIM, and 0.00085 MSE after 2,000 steps, confirming correct token ordering, decoding, and supervision.

Q2.2 · full training
The best checkpoint occurred at epoch 99 with 16.12 dB PSNR and 0.5601 SSIM. The gap between train and validation curves remained small.

Target-ray ablation

| Sources | Conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | Correct | 15.19 | 0.5583 |
| 1 | Shuffled | 11.80 | 0.4874 |
| 1 | None | 12.47 | 0.5285 |
| 2 | Correct | 16.12 | 0.5601 |
| 2 | Shuffled | 11.49 | 0.4721 |
| 2 | None | 12.44 | 0.5270 |
| 4 | Correct | 16.86 | 0.5748 |
| 4 | Shuffled | 11.59 | 0.4729 |
| 4 | None | 12.46 | 0.5368 |
With correct rays, adding sources improves PSNR from 15.19 to 16.86 dB. Shuffling target rays costs 3.4–5.3 dB; removing rays also collapses performance near 12.45 dB. The model therefore uses explicit camera conditioning rather than merely copying appearance from the sources.



Q2.2 · training visualization
Source views provide appearance while the target Plücker rays specify the requested view. The prediction recovers the room layout and dominant colors, with remaining blur around fine geometry and occlusion boundaries.




Q2.3 · bonus: model capacity
The controlled compact model halves width (192) and depth (4 layers) while keeping data, optimization, patch size, and 100 epochs fixed. It trains in 19.4 minutes versus 47.4 minutes. The modest quality loss suggests the baseline is over-capacity for this data budget, though its advantage grows from 0.28 dB at one source to 0.60 dB at four.
Multi-view geometry · Mini-VGGT
A frozen DINOv2 ViT-B/14 encodes RGB patches. Eight pairs of geometry blocks alternate between attention within each frame and global attention across every view. Concatenating the final frame and global features preserves local image evidence while exposing cross-view correspondences.
The camera head predicts world-to-camera translation, quaternion, and horizontal/vertical field of view. The dense head reconstructs positive depth with an exponential activation, adds an RGB-guided residual, and predicts confidence ≥ 1.
Q3.1 · single-scene overfit
Loss fell from 5.48 to −0.177. Depth AbsRel improved from 0.619 to 0.035, δ1 rose from 0.208 to 0.957, and rotation / translation errors dropped to 0.25° / 0.46°. A negative final objective is valid because the confidence term includes −0.2 log(confidence).


Ground truth and predicted geometry after single-scene overfitting.
Q3.2 · full training
The model trained on 224 scenes for 1,000 epochs. The best checkpoint was selected by validation loss (0.8321), not by the final epoch. Curves show noisy scene-to-scene supervision but a stable long-run decline.


Ground truth and predicted geometry for four different held-out validation scenes.
Validation · 28 scenes
Test · 28 scenes
| Split | Metric | @ 5° | @ 15° | @ 30° |
|---|---|---|---|---|
| Val | Rotation | 0.1737 | 0.4943 | 0.6851 |
| Val | Translation | 0.0128 | 0.0721 | 0.1804 |
| Val | Joint | 0.0038 | 0.0549 | 0.1605 |
| Test | Rotation | 0.1952 | 0.5183 | 0.7198 |
| Test | Translation | 0.0071 | 0.0526 | 0.1553 |
| Test | Joint | 0.0013 | 0.0333 | 0.1369 |
Depth and relative rotation generalize substantially better than translation direction. Translation is especially sensitive to small-baseline clips and scene ambiguity; because joint pose uses the worse of rotation and translation error, translation dominates the joint metric. Test depth is slightly worse than validation, while rotation AUC is slightly better, indicating that error modes vary by scene composition rather than a uniform generalization gap.
Q3.3 · bonus: number of views
| Views | AbsRel ↓ | δ1 ↑ | Rot AUC 30 ↑ | Trans AUC 30 ↑ | Joint AUC 30 ↑ | Runtime |
|---|---|---|---|---|---|---|
| 2 | 0.1144 | 0.8571 | 0.6857 | 0.2512 | 0.1917 | 10.71 s |
| 4 | 0.1297 | 0.8601 | 0.6558 | 0.2020 | 0.1605 | 12.74 s |
| 8 | 0.1473 | 0.8223 | 0.6851 | 0.1804 | 0.1605 | 12.28 s |
The same eight-view-trained checkpoint was evaluated without fine-tuning. Two views unexpectedly gave the best AbsRel and joint pose AUC, while four gave the best δ1. Higher view counts also evaluate more pixels and many more pose pairs (28 → 168 → 784), often spanning harder baselines. Runtime includes loading and reporting overhead, so it does not isolate the expected quadratic global-attention scaling. A stronger follow-up would train separate models or randomize view count during training.
Takeaways
A first-camera frame and shared scene scale remove arbitrary global pose and units while preserving multi-view geometry.
LVSM’s sharp ablation drop shows that explicit world-space ray queries—not source appearance alone—select the rendered view.
VGGT learns useful depth and rotation, while translation-direction ambiguity remains the dominant held-out pose error.