Assignment 04 · Carnegie Mellon University

Seeing a scene
from every angle.

From camera geometry to novel-view synthesis and multi-view reconstruction: a hands-on exploration of how transformers understand 3D scenes.

ResearcherKashish Gandhi
Course16-825 · Learning for 3D Vision
DatasetScanNet++ iPhone

How do a collection of 2D images become a coherent 3D scene? This project builds the geometry pipeline first, then trains two small multi-view transformers to render unseen views and predict scene geometry.

224 train · 28 val · 28 test
01

From pixels to rays

Each sampled clip is expressed in a shared coordinate frame. Depth unprojection lifts the pixels into 3D; canonicalization anchors the first camera and normalizes the scene scale.

Point cloud and cameras before canonicalization
World-frame unprojectionDepth values are converted to 3D points and transformed into the original world frame, with camera positions shown alongside the scene.
Canonicalized point cloud and cameras
Canonicalized sceneThe same scene is represented relative to the first selected camera, with positions expressed in normalized units.
8views per sample
224²input resolution
1.000centroid distance, four checks
< 0.00013max reprojection error, pixels

Canonical coordinates

Camera-to-world poses are transformed into camera 0's frame. Camera translations, optical-axis depth and 3D points are divided by the distance from the origin to the valid-point centroid. The centroid itself is not subtracted.

T′ᵢ = inverse(T₀) Tᵢ   ·   scale = ‖mean(pvalid)‖

Ray representation

For each pixel, the inverse intrinsic matrix gives a camera-space direction. Rotating it by the camera-to-world rotation and normalizing gives a unit world-space direction. Plücker coordinates store this direction and the moment of the ray.

ray = [d, o × d]

Four sampled training clips passed the provided data check. Largest point reprojection error: 0.0001231 px; largest ray reprojection error: 0.0001079 px. Centroid distance: 1.0 for each clip.

Full data check

The complete output contains the four sample-level checks and batch shapes.

View the full check_data.py output ↗

Canonicalization check · 036bce3393_frame_004470

For the scene shown above, camera 0 is at the origin to numerical precision. The valid-point centroid is one normalized unit from the origin.

Camera 0 center: [0.000000, −2.38 × 10⁻¹⁶, −1.19 × 10⁻¹⁶]
Valid-point centroid: [−0.416928, −0.387254, 0.822317]
Centroid distance: 0.99999994
02

Render a view, one token at a time.

Mini-LVSM uses source RGB and camera rays to predict target RGB. Target queries contain camera rays only; the target image is never given to the model.

Model at a glance

Images + rays → patches → transformer → RGB

Source images and Plücker rays are patchified and projected to tokens. For each target view, its ray-query tokens attend jointly to all source tokens, then an RGB head decodes the target patches.

Can the model memorize a batch?

Overfit check passed
36.79 dBfinal fixed-batch PSNR
0.9377fixed-batch SSIM
2,000optimization steps
2source views
Source views, target ground truth and LVSM overfit predictions
Fixed-batch overfitEach row shows two source views followed by the ground-truth target and the predicted target. The prediction reproduces the scene layout and viewpoint with some blur and texture differences.

What do the camera rays contribute?

Validation ablation
Source viewsTarget raysPSNR ↑SSIM ↑Target images
1Correct15.050.5578196
1Shuffled11.750.4913196
1None12.380.5428196
2Correct15.870.5562168
2Shuffled11.430.4743168
2None12.340.5392168
4Correct16.520.5697112
4Shuffled11.580.4772112
4None12.370.5498112

With correct target rays, PSNR improves from 15.05 to 16.52 dB as source views increase from one to four. Shuffled and zeroed rays perform worse, indicating that the model uses target-camera information. Source-count comparisons use different numbers of target images, so the averages are not over identical target sets.

Training: 100 epochs, two source views, batch size 1, 224 × 224, patch size 8, width 384, eight transformer layers, learning rate 0.0002, bfloat16 AMP disabled; NVIDIA Tesla T4. Final epoch validation: 15.87 dB PSNR, 0.5562 SSIM. The ablation uses the best checkpoint.

Training dynamics

LVSM training and validation loss, PSNR, and validation SSIM over 100 epochs
Full training: 100 epochsTraining and validation loss and PSNR, with validation SSIM. Curves are generated from the training log.
LVSM training visualization at epoch 95
Training visualization · epoch 95Source views, target ground truth and predicted target, in the order saved by the training script.

Target-ray conditioning

The same two-source-view evaluation setting is shown with the correct target rays, rays shuffled between targets, and zero target rays.

Evaluation with correct target rays
Two views · correct raysPSNR 15.87 dB · SSIM 0.5562
Evaluation with shuffled target rays
Two views · shuffled raysPSNR 11.43 dB · SSIM 0.4743
Evaluation with zero target rays
Two views · no target raysPSNR 12.34 dB · SSIM 0.5392

Additional evaluation visualizations: 1 view, correct · 4 views, correct.

03

Predict the structure behind the views.

Mini-VGGT takes RGB images and predicts cameras and depth. Alternating frame and global attention lets each view reason locally and exchange information across the scene.

Model at a glance

RGB → DINOv2 → frame / global attention → cameras + depth

A frozen DINOv2 encoder produces patch features. A camera head predicts pose and field of view; a depth head predicts per-pixel depth and confidence.

Fixed-clip overfit

Completed
0.0401depth AbsRel ↓
0.9504depth δ1 ↑
0.185°mean rotation error ↓
0.412°mean translation error ↓

These metrics are measured on the fixed training clip used for overfitting, not on held-out scenes. Training used 40 epochs, 25 steps per epoch, batch size 1 and eight views.

VGGT fixed-clip overfit total, camera and depth loss curves
Fixed-clip overfit loss curvesTotal, camera and depth loss over the overfit run.
Ground-truth camera and depth geometry for the fixed training clip
Ground-truth geometryScene 036bce3393, fixed overfit training clip.
Predicted camera and depth geometry for the fixed training clip
Predicted camera + depthThe reconstructed room preserves the overall layout, with some differences in fine details and depth.

Generalization to unseen scenes

Evaluation complete
Current VGGT full training loss curve
Full training loss curveTraining was stopped early near epoch 925; the best validation checkpoint was evaluated on both held-out splits.

Held-out results

MetricValidationTest
Depth AbsRel ↓0.14210.1494
Depth δ1 ↑0.83320.8259
Rotation AUC @ 5° ↑0.18420.2153
Rotation AUC @ 15° ↑0.50600.5477
Rotation AUC @ 30° ↑0.69520.7334
Translation AUC @ 5° ↑0.01760.0105
Translation AUC @ 15° ↑0.08630.0809
Translation AUC @ 30° ↑0.19930.2009
Joint pose AUC @ 5° ↑0.00330.0026
Joint pose AUC @ 15° ↑0.06100.0591
Joint pose AUC @ 30° ↑0.17560.1803

Full validation and test splits, 28 scenes each, using the same best-validation checkpoint.

Depth accuracy is similar across validation and test (AbsRel 0.1421 vs. 0.1494). Translation-direction AUC is substantially lower than rotation AUC, indicating that relative translation remains more difficult on these held-out scenes.

Validation geometry results

Compare ground-truth and predicted cameras and depth on the held-out scenes.

Validation metrics · Test metrics

Held-out scene comparisons

Scene 09c1414f1b

Ground-truth geometry for validation scene 09c1414f1b
Ground truthEight sampled views.
Predicted geometry for validation scene 09c1414f1b
Predicted camera + depthThe overall room and window structure is recognizable, while the table and chair region is more diffuse and depth boundaries are smoother.

Scene 21d970d8de

Ground-truth geometry for validation scene 21d970d8de
Ground truthEight sampled views.
Predicted geometry for validation scene 21d970d8de
Predicted camera + depthLarge scene structures are present, while the predicted depth map has smoother transitions and reduced object detail. The plot viewpoints differ slightly.

Scene 280b83fcf3

Ground-truth geometry for validation scene 280b83fcf3
Ground truthEight sampled views.
Predicted geometry for validation scene 280b83fcf3
Predicted camera + depthThe hallway structure is recognizable, but the prediction produces smoother depth and differs around wall and doorway boundaries.

Scene 2b1dc6d6a5

Ground-truth geometry for validation scene 2b1dc6d6a5
Ground truthEight sampled views.
Predicted geometry for validation scene 2b1dc6d6a5
Predicted camera + depthThe prediction captures the wall and chair region, with smoother depth and differences in reconstructed geometry. The plot viewpoints differ.

Required evaluation: eight views from the first 32 prepared frames, image size 224, seed 825. Run both full splits without a batch limit.