Multi-View Transformers¶

16-825 Learning for 3D: Assignment 4

Author: Benny Zhu (haozhenz)
Date: November 6th 2026

Table of Contents¶

  • 1. Multi-View Data Preprocessing (20 Points)
    • 1.1–1.2 Intrinsics Resizing & Depth Unprojection
    • 1.3 Camera and Depth Canonicalization
    • 1.4–1.5 Camera Rays & Plücker Coordinates
    • 1.7 Final Check
  • 2. Novel-View Synthesis with Mini-LVSM (40 Points)
    • 2.1 Overfitting (10 points)
    • 2.2 Training and Evaluation (30 points)
  • 3. Multi-View Geometry with Mini-VGGT (40 Points)
    • 3.1 Overfitting (10 points)
    • 3.2 Training and Evaluation (30 points)
  • (Bonus) 2.3 Mini-LVSM Analysis and Exploration (10 points)
  • (Bonus) 3.3 Mini-VGGT Analysis and Exploration (10 points)

1. Multi-View Data Preprocessing (20 Points)¶

1.1–1.2 Intrinsics Resizing & Depth Unprojection¶

python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000 --output outputs/unprojection

Unprojection (original world frame, meters):

Scene 1
Unprojection A

1.3 Camera and Depth Canonicalization¶

python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization

Canonicalized geometry (camera 0 = identity):

Scene 1
Canonicalization A
Scene Camera 0 center Valid-point centroid distance
Scene 1 (0, 0, 0) 1.0

1.7 Final Check¶

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

Output:

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.591897105565295e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001089862416847609,
      "max_ray_reprojection_error_px": 0.00010141761595150456,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
} 

Canonicalization: Express all camera poses relative to a reference frame. Typically the first camera is placed at the origin with identity rotation, and every other camera is rewritten relative to it, often with scale normalized as well. The model then sees only relative camera geometry, independent of the scene’s absolute position and scale.

World-space ray directions: For each pixel (u, v), unproject with the inverse intrinsics to get a camera-frame direction K⁻¹[u, v, 1]ᵀ, rotate it into the world frame with the camera-to-world rotation R, and normalize: d = normalize(R K⁻¹[u, v, 1]ᵀ). This is the direction of the ray leaving the camera center through that pixel.

Plücker moment: With camera center o, the moment is m = o × d. Together, (d, m) is a 6D representation of the ray. Replacing o with any other point on the ray leaves m unchanged, so the pair identifies the line itself: d encodes its orientation and m its offset from the origin.


2. Novel-View Synthesis with Mini-LVSM (40 Points)¶

2.1 Overfitting (10 points)¶

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit

Final overfit PSNR: 30.873026529947918

LVSM overfit

2.2 Training and Evaluation (30 points)¶

python -m models.lvsm.train --config configs/lvsm.json
python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none

Training Settings¶

Setting Value
num_source_views 2
Num views / sampling range 8 / 32
Batch size 4
Epochs 100
Learning rate 2e-4
Patch size / width / depth 8 / 384 / 8 layers
AMP (bf16) True

Training Curves¶

LVSM train vis

Ablation Results (Validation)¶

Source views Conditioning PSNR ↑ SSIM ↑
1 correct 15.19246549022441 0.5581682409558978
1 shuffled 11.808103288922991 0.4875379484526965
1 none 12.474374420788823 0.5274129059849954
2 correct 16.12853331792922 0.5601188568841844
2 shuffled 11.480516342889695 0.47215169952029273
2 none 12.45132791428339 0.526719229561942
4 correct 16.854456220354354 0.5747556941849845
4 shuffled 11.587574277605329 0.47292416436331614
4 none 12.45746558053153 0.5379406639507839

Visualizations¶

Training:

Epoch 0:

LVSM train vis

Epoch 95:

LVSM train vis

Evaluation:

Correct conditioning Shuffled conditioning No conditioning
1 source view S1 correct S1 shuffled S1 none
2 source views S2 correct S2 shuffled S2 none
4 source views S4 correct S4 shuffled S4 none

Discussion¶

Effect of the number of source views: Increasing the number of source views from 1 to 2 to 4 gives the model more overlapping observations of the scene, which reduces ambiguity about occluded or unseen regions and generally sharpen predictions, closer to GT.

Effect of ray-conditioning: With correct target rays, the model is told exactly which viewpoint to render and should reconstruct an image close to GT. With shuffled rays, the model is given a different view's camera pose, so a well-behaved model should render content resembling that wrong viewpoint rather than the true target. With no conditioning, the model has no information about which view to synthesize and tends to fall back to a blurry, low-frequency average of plausible content, since it cannot commit to any specific viewpoint.


3. Multi-View Geometry with Mini-VGGT (40 Points)¶

3.1 Overfitting (10 points)¶

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

Overfit loss curve (total / camera / depth):

VGGT overfit loss

GT vs. Predicted geometry:

GT Prediction
Overfit GT Overfit Pred

3.2 Training and Evaluation (30 points)¶

python -m models.vggt.train --config configs/vggt.json --output outputs/vggt_train
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/metrics.json --visualization outputs/vggt_train/visualization/index.html
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split test --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/test_metrics.json

Training Settings¶

Setting Value
num_views / sampling_range 8 / 32
Batch size 16
Epochs 1000
Learning rate 6e-4
Encoder Frozen DINOv2 ViT-B/14
Aggregator 8 frame/global pairs, width 384, 6 heads
Loss weights (camera / depth) camera=5.0,depth=1.0

Training Loss Curve (total / camera / depth)¶

VGGT loss

Evaluation Metrics¶

Metric Val Test
Sample counts
Num. samples (scenes) 28 28
Num. valid depth pixels 10,347,720 10,517,881
Num. pose pairs 784 784
Num. degenerate pose pairs 0 0
Loss
Total loss 0.8234 0.7552
Camera loss 0.1081 0.0902
Depth loss 0.2829 0.3043
Depth accuracy
AbsRel ↓ 0.155 0.169
RMSE (normalized) ↓ 0.203 0.209
δ1 ↑ 0.800 0.767
Raw pose error (mean, degrees)
Rotation error (°) ↓ 12.32 8.82
Translation error (°) ↓ 39.84 41.48
Rotation AUC ↑
@3° 0.091 0.115
@5° 0.178 0.210
@15° 0.508 0.547
@30° 0.699 0.747
Translation AUC ↑
@3° 0.0085 0.0084
@5° 0.0156 0.0099
@15° 0.0880 0.0683
@30° 0.2082 0.1639
Joint Pose AUC ↑
@3° 0.0000 0.0009
@5° 0.0018 0.0026
@15° 0.0645 0.0516
@30° 0.1815 0.1517

GT vs. Predicted Geometry (4 validation scenes)¶

Scene GT Prediction
Scene 1 GT1 Pred1
Scene 2 GT2 Pred2
Scene 3 GT3 Pred3
Scene 4 GT4 Pred4

Discussion¶

Across all four held-out validation scenes, the predicted camera layout closely matches GT. In Scenes 1, 3 and 4, the relative camera cluste is reconstructed in the same spatial arrangement as GT.

The point cloud geometry, however, degrades more visibly than the camera poses. In every scene, the predicted structure is noticeably noisier and less sharply defined than GT — flat surfaces and edges (e.g. the shelf unit in Scene 2, the desk in Scene 4) appear "smeared" or ballooned outward rather than forming crisp planar structure. Scene 4 shows the largest degradation: GT shows a well-defined desk/room boundary, while the prediction spreads into a much more diffuse, scattered gray cloud, even though the camera trajectory above it is still well localized. This matches the quantitative depth error (AbsRel ≈ 0.16–0.17, i.e. roughly 15–17% relative depth error) — a moderate but non-trivial error that manifests visually as exactly this kind of loss of fine geometric detail rather than gross failure.

Taken together, this suggests the model has learned to generalize the global scene structure and camera arrangement well, but still struggles to generalize precise, per-pixel depth to unseen scenes — likely because depth regression is a harder, more scene-specific task than relative camera pose, and the model was trained for a comparatively short schedule relative to the diversity of ScanNet++ scenes.


(Bonus) 2.3 Mini-LVSM Analysis and Exploration (10 points)¶

Question: Does adding an L1 and SSIM loss term on top of the baseline MSE photometric loss improve the PSNR, SSIM, and visual sharpness of Mini-LVSM's synthesized views?

Experimental settings: Train with the same configuration as Q2.2 (100 epochs, S=2 source views), but with --aux-loss "l1=0.1,ssim=0.1" enabled, then evaluate with the same source-view/conditioning ablation as Q2.2 and compare PSNR/SSIM/visual sharpness against the MSE-only baseline..

Source views Conditioning Baseline PSNR +L1+SSIM PSNR Baseline SSIM +L1+SSIM SSIM
1 correct 15.19 14.76 0.5582 0.5719
2 correct 16.13 15.30 0.5601 0.5743
4 correct 16.85 15.55 0.5748 0.5870
Baseline (S=2) Variant (S=2)
Baseline Variant

Conclusions: Adding L1 and SSIM loss terms alongside MSE consistently decreases PSNR while consistently increasing SSIM across every evaluated setting. This is a coherent trade-off: MSE is minimized by hedging pixel values toward a blurry average, which maximizes PSNR but produces soft, low-structure predictions; SSIM explicitly rewards local structural/contrast consistency rather than exact per-pixel values, so optimizing for it pulls the model away from the PSNR-optimal blurry solution toward sharper, more structured outputs. This illustrates that PSNR and structural quality are not the same objective, and improving one can come at the cost of the other.