Assignment 4: Multi-View Transformers

Parth Burujwale Andrew ID: pburujwa 16-825: Learning for 3D Vision

This assignment explores multi-view geometry preprocessing, novel-view synthesis with Mini-LVSM, and camera/depth reconstruction with Mini-VGGT using ScanNet++ iPhone images.


1. Multi-View Data Preprocessing

1.1. Resizing Camera Intrinsics

I updated the cropped camera intrinsics to match the resized image dimensions. Focal lengths scale independently along the horizontal and vertical axes.

The dataloader already handles the center-crop adjustment.

1.2. Depth Unprojection

For each pixel, I computed its camera-space point using:

\[ \mathbf{X}_{c} = zK^{-1}[u,v,1]^T. \]

The camera-to-world pose then transforms the point:

\[ \mathbf{X}_{w} = R\mathbf{X}_{c} + \mathbf{t}. \]

The supplied depth is optical-axis Z-depth, so the backprojected camera vectors are not normalized before multiplication by depth. The provided pipeline masks invalid output points.

Unprojection in original world coordinates

Unprojected RGB-D points and camera poses in the original world coordinate system, measured in meters.

1.3. Camera and Depth Canonicalization

I expressed every camera pose in the first sampled camera’s coordinate frame:

\[ T_{\mathrm{reference}\leftarrow i} = T_{\mathrm{world}\leftarrow 0}^{-1} T_{\mathrm{world}\leftarrow i}. \]

I then used the provided scene-scale calculation:

\[ s = \left\| \operatorname{mean}_{\mathrm{valid}} (\mathbf{X}_{\mathrm{reference}}) \right\|_2. \]

Camera 0 remains at the origin, and the valid-point centroid has distance approximately one from it.

Canonicalized cameras and points

Geometry in the first camera’s frame with shared scale normalization. The submission screenshot shows the first camera center and valid-point centroid distance.

1.4. World-Space Camera Rays

Each ray originates at the camera center. Its direction is obtained by backprojecting the pixel through the intrinsics, rotating into the canonical world frame, and normalizing:

\[ \mathbf{o} = \mathbf{t}, \qquad \mathbf{d} = \frac{RK^{-1}[u,v,1]^T} {\left\|RK^{-1}[u,v,1]^T\right\|_2}. \]

Translation determines the origin, while rotation determines the world-space direction.

1.5. Plucker Coordinates

I represented each ray using six channels:

\[ [\mathbf{d},\mathbf{m}], \qquad \mathbf{m} = \mathbf{o}\times\mathbf{d}. \]

The direction specifies the ray orientation. The moment encodes the line’s location relative to the origin and remains unchanged when the origin moves along the same line.

1.6. Dataloader

The dataset contains 224 training, 28 validation and 28 test scenes.

The training loader randomly samples eight views from a sampling window and applies the implemented preprocessing functions. The first sampled view defines the reference camera.

Validation and test use prepared geometry and fixed view selection. RGB remains valid independently of the depth-validity mask.

1.7. Geometry Verification

All four checked training samples satisfied the required reprojection threshold of 0.01 pixels.

Sample Valid depth fraction Maximum point error (px) Maximum ray error (px) Centroid distance
e050c15a8d_frame_007080 0.977913 0.00010845 0.00009414 1.0
5f99900f09_frame_002820 0.990312 0.00012302 0.00010790 1.0
47b37eb6f9_frame_002720 0.960995 0.00009537 0.00008217 1.0
4c5c60fa76_frame_006690 0.751672 0.00010784 0.00009537 1.0
Geometry check output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010844774078577757,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.5367431640625e-05,
      "max_ray_reprojection_error_px": 8.217109279939905e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010783717152662575,
      "max_ray_reprojection_error_px": 9.53696362557821e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1. Fixed-Batch Overfitting

I trained on one fixed batch for 2,000 optimization steps.

Final fixed-batch metric Result
MSE 0.00086
PSNR 30.95 dB
SSIM 0.8706
Runtime 240 seconds

The final PSNR exceeded the approximately 30 dB sanity-check target. This demonstrates that the model can memorize the fixed batch.

Mini-LVSM fixed-batch overfit

Columns: two source views, target ground truth, and prediction.

2.2. Full Training

Setting Value
Training scenes 224
Image resolution 224 × 224
Sampled views 8
Source / target views 2 / 6
Batch size 4
Epochs 100
Patch size 8
Transformer width 384
Transformer blocks 8
Parameters Approximately 14.6 million
Optimizer AdamW
Learning rate 0.0002
Weight decay 0.05
Adam betas 0.9, 0.95
Warmup 200 steps
Gradient clipping 1.0
Precision bfloat16 autocast
Hardware NVIDIA H100 80 GB
Seed 825
Mini-LVSM training curves

Training and validation MSE and PSNR, together with validation SSIM.

Training and validation improved before approaching a plateau. The best validation PSNR was 16.1409 dB at epoch 91, with SSIM 0.5601.

Mini-LVSM normal-training visualization

Normal-training visualization showing source images, target ground truth and predicted RGB.

Evaluation Ablations

I evaluated best.pt with 1, 2 and 4 source views under three target-ray conditions:

Sources Target rays PSNR ↑ SSIM ↑ MSE ↓ Target images
1 Correct 15.19 0.5571 0.03523 196
1 Shuffled 11.79 0.4862 0.07307 196
1 None 12.46 0.5297 0.05921 196
2 Correct 16.14 0.5601 0.02927 168
2 Shuffled 11.47 0.4719 0.07824 168
2 None 12.42 0.5264 0.05960 168
4 Correct 16.87 0.5757 0.02438 112
4 Shuffled 11.58 0.4746 0.07848 112
4 None 12.45 0.5357 0.05905 112

Source-View Comparisons

One source with correct rays
Two sources with correct rays
Four sources with correct rays

Each row contains the source views followed by target ground truth and prediction.

Correct-ray PSNR increased from 15.19 dB with one source to 16.87 dB with four sources. Additional source observations can provide more scene information. However, the evaluation target sets also change with source count, so these results are not a comparison on identical target images.

Target-Ray Conditioning

Two sources with shuffled rays
Two sources without target rays

With two sources, shuffling rays reduced PSNR by 4.67 dB, while removing rays reduced it by 3.72 dB. Correct target-camera information is therefore important to the model’s predictions.

Shuffled rays produced layouts inconsistent with the target. With zero target rays, predictions were nearly identical across target views because the model no longer received distinct target-camera queries.

Even with correct rays, held-out predictions remained blurry and lost furniture details. The strong fixed-batch overfit result did not translate into equally detailed reconstruction of unseen scenes.


3. Multi-View Geometry with Mini-VGGT

3.1. Single-Clip Overfitting

I trained on one fixed eight-view training clip to verify that the model could fit camera poses and depth.

Final fixed-clip metric Result
Depth AbsRel ↓ 0.0371
Depth δ1 ↑ 0.9549
Mean rotation error ↓ 0.1298°
Mean translation-direction error ↓ 0.2404°
Joint pose AUC @ 5° ↑ 0.9929
Mini-VGGT overfit loss curves

Total, camera and depth losses decrease substantially during single-clip fitting.

The depth objective can become negative because it includes the confidence term \(-0.2\log(c)\). A negative total loss is therefore possible.

Ground truth Predicted camera + depth
Overfit ground truth Overfit prediction

The predicted camera arrangement and broad room structure closely matched the fixed clip’s ground truth. Predicted depth was smoother and retained less fine detail.

3.2. Full Training

Setting Value
Training scenes 224
Input RGB only
Image resolution 224 × 224
Views per sample 8
Sampling range 32
Batch size 16
Configured training epochs 1,000
Encoder Frozen DINOv2 ViT-B/14
Aggregation 8 frame/global pairs
Transformer width 384
Attention heads 6
Camera-head transformer blocks 4
Learning rate 0.0006
Weight decay 0.01
Warmup 10 epochs
Camera / depth loss weights 5.0 / 1.0
Hardware NVIDIA H100 80 GB
Best-checkpoint selection Lowest validation loss
Mini-VGGT full-training loss curves

Training and validation total, camera and depth losses.

Both training and validation losses decreased, with smaller improvements late in training. Validation loss did not show a sustained upward trend.

Validation and Test Evaluation

I evaluated the checkpoint selected by validation loss on the complete validation and test splits.

Both evaluations used 8 views from the first 32 prepared frames, 224 × 224 images and seed 825, without a maximum-batch restriction.

Metric Validation Test
Scenes 28 28
Depth AbsRel ↓ 0.1479 0.1493
Depth δ1 ↑ 0.8161 0.8248
Rotation AUC @ 5° ↑ 0.1972 0.2089
Rotation AUC @ 15° ↑ 0.5293 0.5475
Rotation AUC @ 30° ↑ 0.7126 0.7298
Translation AUC @ 5° ↑ 0.0179 0.0122
Translation AUC @ 15° ↑ 0.0834 0.0772
Translation AUC @ 30° ↑ 0.1924 0.1819
Joint pose AUC @ 5° ↑ 0.0046 0.0036
Joint pose AUC @ 15° ↑ 0.0579 0.0576
Joint pose AUC @ 30° ↑ 0.1662 0.1623

AUC values are reported on a 0–1 scale.

Geometry Comparisons on Four Validation Scenes

Scene 1: 09c1414f1b

Ground truth Predicted camera + depth
Scene 1 ground truth Scene 1 prediction

The prediction retained broad scene structure but blurred furniture boundaries. The predicted cameras were more tightly clustered than in ground truth.

Scene 2: 21d970d8de

Ground truth Predicted camera + depth
Scene 2 ground truth Scene 2 prediction

The broad room layout remained recognizable, while surfaces were smoother and some structural boundaries were distorted.

Scene 3: 280b83fcf3

Ground truth Predicted camera + depth
Scene 3 ground truth Scene 3 prediction

The predicted hallway appeared compressed and curved relative to ground truth, with reduced doorway detail.

Scene 4: 2b1dc6d6a5

Ground truth Predicted camera + depth
Scene 4 ground truth Scene 4 prediction

The prediction preserved the rough layout but showed smeared chair/desk geometry and overlapping surfaces.

Discussion

Depth performance was similar on validation and test, with AbsRel near 0.15 and δ1 above 0.81 on both splits.

Translation-direction estimation remained substantially weaker than rotation estimation, limiting joint pose AUC. The qualitative examples also showed camera-arrangement differences and imperfect alignment of points reconstructed across views.

Compared with single-clip overfitting, held-out reconstructions lost fine detail and exhibited greater distortion. The model recovered coarse scene structure, but depth smoothing and camera-pose errors limited detailed multi-view consistency.