Project 4: Multi-View Transformers

By Teja Kanthamneni · 16-825 Learning for 3D Vision

Preparing multi-view geometry, synthesizing new views with Mini-LVSM, and predicting cameras and depth with Mini-VGGT.

Part 1 · Multi-view geometry

1. Data Preprocessing

I implemented the geometry functions used to prepare RGB, depth, cameras, and rays from the ScanNet++ training scenes. Images are center-cropped and resized to 224 × 224. The cameras use OpenCV coordinates: +X to the right, +Y down, and +Z forward.

1.1–1.2 Intrinsics and depth unprojection

Resizing changes the focal lengths and principal point. We use the half-pixel convention so that the resized camera matches the image resize. For unprojection, each pixel is converted to a camera-space direction using the inverse intrinsic matrix, multiplied by its z-depth, and transformed into world coordinates using the camera pose.

The following screenshots show two training scenes in their original world coordinates, measured in meters. The red frustum marks the first sampled camera.

Original-world points and cameras for training scene 036bce3393
Scene 036bce3393 · after depth unprojection
Original-world points and cameras for training scene 076c822ecc
Scene 076c822ecc · after depth unprojection

1.3 Canonicalization

Each sample is expressed in the coordinate frame of its first selected camera. We left-multiply every pose by the inverse of camera 0’s original pose, then divide camera translations, depth, and 3D points by the distance of the valid-point centroid from the origin. This makes camera 0 identity and the centroid distance 1.

This removes the arbitrary world frame and scene scale, giving both models a consistent convention for each sample. The same frames are shown below after canonicalization.

Canonicalized points and cameras for scene 036bce3393; camera zero at origin and centroid distance one
Scene 036bce3393 · camera 0 center ≈ (0, 0, 0); valid-point centroid distance ≈ 1.000; original scene scale 1.867 m.
Canonicalized points and cameras for scene 076c822ecc; camera zero at origin and centroid distance one
Scene 076c822ecc · camera 0 center ≈ (0, 0, 0); valid-point centroid distance ≈ 1.000; original scene scale 2.830 m.

1.4–1.6 Rays, Plücker coordinates, and sampling

A ray starts at the camera center. Its world-space direction is the normalized camera-space pixel direction rotated by the camera-to-world rotation. Translation changes the ray origin, while rotation changes its direction.

We represent each ray as [d, o × d]. The direction d describes where the ray points, and the moment o × d describes the line’s position relative to the origin. Moving the chosen origin along the ray leaves the moment unchanged.

During training, the loader samples eight frames in random order from a 32-frame window. The first selected view becomes camera 0. Validation and test instead use fixed prepared frames and the provided geometry, so they do not call my training preprocessing functions.

1.7 Final data check

I checked four samples from the five-scene local training subset. Each sample has eight views at 224 × 224. All reprojection errors are well below the required 0.01 pixels, and all valid-point centroid distances are approximately 1.

Training sceneValid depthPoint error (px)Ray error (px)Centroid distance
07ff1c45bb98.26%0.0000920.0000871.00000000
07f5b601ee85.46%0.0001110.0001170.99999994
036bce339396.30%0.0001070.0001021.00000000
076c822ecc93.31%0.0001380.0001331.00000000
Show the complete data-check output
{
  "split": "train",
  "dataset_samples": 5,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "07ff1c45bb_frame_002580",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9826286435127258,
      "max_reprojection_error_px": 9.184284863295034e-05,
      "max_ray_reprojection_error_px": 8.699346653884277e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "07f5b601ee_frame_006080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.8545569777488708,
      "max_reprojection_error_px": 0.00011122578143840656,
      "max_ray_reprojection_error_px": 0.00011726274533430114,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "036bce3393_frame_006330",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9630177021026611,
      "max_reprojection_error_px": 0.0001068115234375,
      "max_ray_reprojection_error_px": 0.00010235906665911898,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "076c822ecc_frame_001080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.933085560798645,
      "max_reprojection_error_px": 0.00013762763410340995,
      "max_ray_reprojection_error_px": 0.00013306154869496822,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Part 2 · Novel-view synthesis

2. Mini-LVSM

The source inputs contain RGB and Plücker rays. Target queries contain only the target rays. I patchify and project the inputs, build one sequence per target view, and use full attention over the source tokens and that target’s query tokens. The RGB head decodes the target tokens into the predicted image. Target RGB is used only for supervision.

2.1 Overfitting one fixed batch

I trained on one fixed batch for 2,000 optimizer steps using MSE loss. The final overfit PSNR is 30.885 dB. SSIM is 0.8705, and MSE is 0.000873.

The overfit predictions recover the room layout and object colors much more closely than the full-split model. Fine texture is still less clear than the ground truth, but this check shows that the model and loss can fit a fixed batch.

2.2 Full training and evaluation

Training detailSetting
Data224 training scenes; 28 validation scenes; 224 × 224 RGB
Views8 per sample; 2 source views and 6 targets; sampling range 32
Model8 transformer blocks; width 384; 6 attention heads; patch size 8
OptimizerAdamW; learning rate 0.0002; weight decay 0.05; betas (0.9, 0.95)
Schedule100 epochs; batch size 4; 200-step warmup, then cosine decay to 10% of peak LR
Loss and precisionMSE; bfloat16 autocast, float32 loss; gradient clipping at 1.0
CheckpointBest validation PSNR; saved epoch 99 (the 100th epoch)
Evaluation8 fixed views from the first 32 prepared frames; seed 825
Mini-LVSM training and validation MSE and PSNR, and validation SSIM over 100 epochs
Training curves · displayed epochs are numbered 1–100.

The validation PSNR increases from 12.25 dB to 16.14 dB, while SSIM increases from 0.3844 to 0.5600. The curves improve throughout training, but the held-out images remain much blurrier than the overfit result.

Training visualizations

These are validation images saved during training. The later predictions recover more of the scene’s broad color arrangement, but object boundaries and textures remain soft. The visible patch structure is also much stronger in the prediction than in the ground truth.

Source-view and target-ray ablations

I evaluated the same best checkpoint with 1, 2, and 4 source views. Correct uses the true target rays, shuffled uses another target view’s rays, and zero rays removes the target-camera information.

Source viewsTarget raysPSNR (dB) ↑SSIM ↑Target images
1Correct15.2080.5583196
1Shuffled11.8080.4872196
1Zero rays12.4620.5269196
2Correct16.1400.5600168
2Shuffled11.4880.4717168
2Zero rays12.4310.5263168
4Correct16.8800.5748112
4Shuffled11.5860.4724112
4Zero rays12.4450.5372112

With correct rays, PSNR increases from 15.208 dB with one source to 16.880 dB with four. More source views provide additional coverage of the scene, although fine detail is still poorly reconstructed. The target sets are not identical across these settings: increasing the source count leaves fewer target views, giving 196, 168, and 112 evaluated target images, respectively.

Incorrect rays have a larger effect than the source-view count. With two sources, shuffling the rays reduces PSNR from 16.140 to 11.488 dB, while removing them gives 12.431 dB. Wrong rays can be worse than no rays because they ask the model to synthesize a different viewpoint. Adding sources does not consistently help when the target rays are wrong or missing.

2.3 Adding an SSIM Loss

MSE versus MSE + 0.1 × (1 − SSIM)

I tested whether adding a structural loss improves the synthesized images. The second run uses MSE + 0.1 × (1 − SSIM), keeping the architecture, data settings, seed, optimizer, learning-rate schedule, and 100-epoch budget the same. Both checkpoints are selected by validation PSNR. Validation MSE is still computed without the SSIM penalty, so it can be compared directly.

Source viewsBaseline PSNRSSIM-loss PSNRBaseline SSIMSSIM-loss SSIM
115.20814.8520.55830.5737
216.14015.4920.56000.5759
416.88015.9000.57480.5897
Validation PSNR, SSIM and MSE for baseline and SSIM-loss Mini-LVSM runs
Both runs use two source views and correct rays; epochs in this saved plot are zero-based (0–99).

The SSIM term improves SSIM for all three source-view counts, but lowers PSNR. With two sources, SSIM increases from 0.5600 to 0.5759, while PSNR decreases from 16.140 to 15.492 dB. MSE also increases from 0.02938 to 0.03248. The added loss changes the balance between pixel accuracy and local structure; it does not improve both metrics together.

The SSIM run looks smoother and has less obvious patch structure in this example, but the table, chairs, and other objects are still blurred. The higher structural score does not mean that the model has recovered their fine details. This is one run per objective, so it also does not establish how much the difference varies across random seeds or other loss weights.

Full SSIM-run ablation metrics
Source viewsTarget raysPSNR (dB) ↑SSIM ↑Target images
1Correct14.8520.5737196
1Shuffled11.9030.5142196
1Zero rays12.4320.5442196
2Correct15.4920.5759168
2Shuffled11.7010.5063168
2Zero rays12.4070.5404168
4Correct15.9000.5897112
4Shuffled11.8470.5137112
4Zero rays12.4340.5503112

Part 3 · Multi-view reconstruction

3. Mini-VGGT

Mini-VGGT predicts cameras and depth from RGB alone. A frozen DINOv2 encoder produces patch features. Frame attention processes each image independently, and global attention allows tokens from all views of the same scene to interact. The final frame and global features are concatenated for the prediction heads.

The camera head predicts translation, a quaternion, and two fields of view. The depth head predicts pixels within each patch and refines the assembled depth using RGB. I supervised the cameras using translation, quaternion, and FoV losses, and depth using regression, confidence, and multi-scale gradient losses.

3.1 Overfitting one fixed clip

I trained on one fixed eight-view clip for 40 epochs, with 25 updates per epoch and learning rate 0.001. This is 1,000 optimizer steps on the same clip. Although the configured batch size is 16, the fixed dataset contains only one example, so the actual overfit batch has one clip.

MetricBefore trainingAfter 40 epochs
Depth AbsRel ↓0.61860.0384
Depth δ1 ↑0.20830.9527
Mean rotation error ↓49.73°0.19°
Mean translation-direction error ↓90.00°0.44°
Mini-VGGT total, camera and depth overfit losses over 40 epochs
Fixed-clip overfit curves.

The losses decrease throughout the run, and camera errors become small. The final total loss is −0.1519. A negative loss is possible here because the confidence objective contains −0.2 log(confidence).

The low pose errors indicate that the camera head has fit this clip closely. Depth is still approximate, with AbsRel 0.0384, so fitting the cameras does not make every surface exact.

3.2 Full training and evaluation

Training detailSetting
Data224 training scenes; 28 validation and 28 test scenes
Views and resolution8 views; sampling range 32; 224 × 224 RGB
EncoderFrozen DINOv2 ViT-B/14
Aggregation8 frame/global pairs; width 384; 6 attention heads
OptimizerAdamW; initial LR 0.0006; weight decay 0.01
Schedule1,000 epochs; batch size 16; 10 warmup epochs, then cosine decay to 5% of peak LR
Loss5 × camera loss + depth loss; camera FoV term weighted by 0.5
Precisionbfloat16 forward pass; gradient clipping at 1.0
CheckpointEpoch 843, selected by lowest validation loss (0.8446)
EvaluationFull val/test splits; 8 views from the first 32 prepared frames; 224px; seed 825
Mini-VGGT train and validation total, camera and depth losses over 1000 epochs
Full-training curves · total loss includes a camera weight of 5; component plots are unweighted.

Validation loss drops quickly early in training and then improves more gradually. The best checkpoint is from epoch 843, rather than the final epoch, whose validation loss is 0.9157. I use the best checkpoint for both evaluation splits and the geometry comparisons.

Validation and test metrics

Lower AbsRel is better. Higher δ1 and AUC are better. AUC values are on a 0–1 scale. Each split contains 28 scenes and 784 valid camera pairs.

Required metricValidationTest
Depth AbsRel ↓0.13760.1539
Depth δ1 ↑0.83910.7977
Rotation AUC @ 5° ↑0.18720.2176
Rotation AUC @ 15° ↑0.50640.5501
Rotation AUC @ 30° ↑0.68980.7415
Translation AUC @ 5° ↑0.01890.0084
Translation AUC @ 15° ↑0.08380.0593
Translation AUC @ 30° ↑0.19130.1582
Joint pose AUC @ 5° ↑0.00460.0023
Joint pose AUC @ 15° ↑0.06260.0462
Joint pose AUC @ 30° ↑0.16580.1463

Depth generalizes reasonably well. AbsRel is 0.1376 on validation and 0.1539 on test, while δ1 is 0.8391 and 0.7977. Test rotation is better than validation rotation at all three AUC thresholds. Translation is less accurate on both splits and is worse on test, which also keeps joint pose AUC low.

The mean rotation errors are 12.88° on validation and 9.11° on test. Mean translation-direction errors are much larger, at 41.67° and 43.25°. Errors in camera placement can make the combined point cloud inconsistent even when individual depth maps capture the room layout.

Geometry on four held-out scenes

Each pair compares GT with the prediction in the same canonical coordinates, with matching camera angle and axis ranges. The point clouds are reconstructed from predicted cameras and depth without GT alignment. As in the provided viewer, GT-valid pixels restrict the displayed points. Black pixels in the depth panels are missing GT measurements.

Scene 09c1414f1b

The dining area has a recognizable overall shape, but the predicted table and chair depths are smoother than GT. Thin chair parts and the table boundary are less distinct. The combined cloud also has thicker, more scattered surfaces, showing that matching the broad room layout is easier than recovering fine geometry.

Scene 21d970d8de

The prediction captures the floor’s depth gradient, but the reconstructed room surfaces are curved and spread out compared with GT. The larger shape is present, while the wall and floor boundaries are less consistent across views.

Scene 280b83fcf3

The corridor’s near-to-far depth change is visible in the prediction. However, the boundary along the left wall is blurred, and the point cloud has thicker, distorted surfaces. Preserving the overall depth ordering does not fully recover the narrow room geometry.

Scene 2b1dc6d6a5

The chair and desk area remain recognizable, and the foreground/background depth separation is similar to GT. The predicted cloud is more scattered, with less consistent alignment between views. This example shows how camera errors can hurt the combined reconstruction even when one depth image looks relatively close.

The single-clip model fits much more closely than the full model on unseen scenes. The held-out results retain depth and pose errors, especially translation. The remaining reconstruction errors should therefore be considered alongside the low overfit losses.