16-825 Assignment 4: Multi-View Transformers

Jerry Hou

November 2026

All experiments were run on a single NVIDIA L4 (24 GB) with bf16 autocast. Image paths are relative to this file.

1. Multi-View Data Preprocessing

1.1–1.2 Unprojection (original world frame, meters)

python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000 --output outputs/unprojection

Depth is z-depth, so each pixel is unprojected as \(X_c = z\,K^{-1}[u, v, 1]^\top\) and mapped to world space with the camera-to-world pose. Points from all 8 views overlap into one consistent room, and each camera frustum sits where it should relative to the geometry it observes. The first camera is at an arbitrary world location: for scene 036bce3393, the camera center is (3.35, 4.68, 1.47) m and the centroid is 6.67 m from the origin.

Unprojection, scene 036bce3393
Unprojection, scene 076c822ecc
Unprojection, scene 079a326597

1.3 Canonicalization (first-camera frame, normalized units)

python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization

These are the same scenes and frames as above. In every scene the first camera center is (0, 0, 0) and the valid-point centroid distance is 1.000. The info panel on the right of each screenshot shows both values.

Scene First camera center Scene scale (m) Centroid distance
036bce3393_frame_004470 (0.0000, −0.0000, 0.0000) 1.867 1.000
076c822ecc_frame_002890 (0.0000, −0.0000, 0.0000) 2.830 1.000
Canonicalization, scene 036bce3393
Canonicalization, scene 076c822ecc
Canonicalization, scene 079a326597

1.7 check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

Sequence Views Res Valid depth Max point reproj. err (px) Max ray reproj. err (px) Centroid dist.
e050c15a8d_frame_007080 8 224 0.978 1.13e−4 1.09e−4 1.0000
5f99900f09_frame_002820 8 224 0.990 1.23e−4 1.11e−4 1.0000
47b37eb6f9_frame_002720 8 224 0.961 9.88e−5 8.90e−5 1.0000
4c5c60fa76_frame_006690 8 224 0.752 1.09e−4 1.03e−4 1.0000

Batch shapes (batch size 2): rgb [2,8,3,224,224], depth/depth_valid [2,8,1,224,224], K [2,8,3,3], T_world_from_camera [2,8,4,4], original_world_from_reference [2,4,4], scene_scale [2], points_world / ray_origins / ray_directions [2,8,3,224,224], plucker [2,8,6,224,224]. All reprojection errors are about 1e−4 px, far below the 0.01 px threshold. The full raw output is in output1-7.txt.

Explanations

Canonicalization. Every pose is left-multiplied by \(T_{\text{ref}}^{-1}\), the inverse of the first sampled camera’s camera-to-world pose. Camera 0 becomes the identity, and all geometry is expressed in its coordinate frame. Then a single scene scale \(s\) is computed: the distance in meters from the origin to the centroid of the valid points. Camera translations, depths and points are all divided by \(s\). Rotations are unchanged, and depth/point consistency is preserved because everything shares the same \(s\). After this step, every training sample has the same gauge: no arbitrary world origin or orientation, and a metric scale near 1. Without it, the network would have to predict poses in an arbitrary world frame, which cannot be determined from the images.

World-space ray directions. For pixel \((u, v)\), the camera-space direction is \(K^{-1}[u, v, 1]^\top\), using the intrinsics after resizing. It is rotated into the canonical world frame with \(R_{wc}\) and then normalized to unit length. The ray origin is the camera center \(t_{wc}\), which is the same for every pixel of a view. Reprojecting origin + depth-scaled direction lands back on the source pixel to within about 1e−4 px.

Plücker moment. A ray is encoded as 6 channels \([\mathbf d, \mathbf m]\) with \(\mathbf m = \mathbf o \times \mathbf d\). The moment is the same for every point on the line: for any point \(\mathbf p = \mathbf o + t\mathbf d\), \(\mathbf p \times \mathbf d = \mathbf o \times \mathbf d\). Its magnitude is the line’s distance from the origin, and its direction is the normal of the plane containing the line and the origin. So \([\mathbf d, \mathbf m]\) identifies the ray as a line, regardless of which point on it is called the origin. It is a smooth per-pixel camera embedding that a transformer can use directly.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit

After 2000 steps on one fixed batch, the train loss fell to 6.9e−3. Final overfit PSNR: 30.90 dB (SSIM 0.870, 24 target images), which clears the 30 dB target.

Overfit visualization (val_epoch000.png). Columns: source views, ground truth, prediction.

2.2 Training and evaluation

Training settings (configs/lvsm.json):

LVSM training curves: MSE loss, train/val PSNR and val SSIM. The SSIM-loss run from Q2.3 is overlaid.

Validation ablations (python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none):

Source views Conditioning PSNR (dB) ↑ SSIM ↑ # images
1 correct 15.20 0.558 196
1 shuffled 11.81 0.488 196
1 none 12.48 0.528 196
2 correct 16.13 0.561 168
2 shuffled 11.50 0.473 168
2 none 12.45 0.526 168
4 correct 16.85 0.576 112
4 shuffled 11.59 0.474 112
4 none 12.46 0.536 112

Training visualizations. Columns: source views, ground truth, prediction.

Validation at epoch 0
Validation at epoch 95

Evaluation visualizations (columns: source views, ground truth, prediction):

S=1, correct rays
S=2, correct rays
S=4, correct rays
S=2, shuffled rays
S=2, no rays

Discussion.

2.3 Bonus: adding an SSIM loss

Question. Does adding a structural loss term to the photometric MSE make predictions sharper and improve SSIM, and what does it cost in PSNR?

Setup. The loss is \(\mathcal L = \text{MSE} + 0.2\,(1 - \text{SSIM})\) (configs/lvsm_ssim.json, ssim_weight: 0.2, implemented in view_synthesis_loss). Everything else matches the baseline: architecture, data, seed, optimizer, schedule, 100 epochs, and 2 training source views. Both models were evaluated on the same validation views with correct rays.

Source views Baseline PSNR +SSIM PSNR Baseline SSIM +SSIM SSIM
1 15.20 14.78 0.558 0.576
2 16.13 15.41 0.561 0.579
4 16.85 15.78 0.576 0.593
+SSIM model, S=2, correct rays (columns: source views, ground truth, prediction)

Results and conclusions.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

Settings: 1 scene, 40 epochs × 25 steps, learning rate 1e−3. All three losses fall smoothly. The camera loss goes 0.34 → 0.0026. The depth loss becomes negative, which is expected: it includes the \(-\log(\text{confidence})\) regularizer, so it goes below zero once the model predicts accurate depth with high confidence. The predicted cameras and point cloud closely match the ground truth.

Overfit loss curves: total, camera and depth
Overfit scene: ground truth
Overfit scene: prediction (predicted camera + depth)

3.2 Training and evaluation

Training settings (configs/vggt.json, run outputs/vggt_train2):

Training loss curves: total, camera and depth, train and validation

Metrics (8 views from the first 32 prepared frames, 224 px, seed 825, full splits of 28 scenes / 784 pose pairs each):

Split AbsRel ↓ δ1 ↑ Rot AUC@5/15/30 ↑ Trans AUC@5/15/30 ↑ Pose AUC@5/15/30 ↑
Val 0.145 0.832 0.214 / 0.539 / 0.717 0.010 / 0.078 / 0.191 0.004 / 0.060 / 0.173
Test 0.159 0.795 0.188 / 0.501 / 0.698 0.013 / 0.076 / 0.194 0.001 / 0.047 / 0.167

Mean errors: rotation 11.8° (val) / 10.5° (test); translation direction 41.2° (val) / 39.1° (test).

GT vs. prediction on four validation scenes (left: ground truth, right: predicted camera + depth):

Ground truth Prediction

Scenes: 09c1414f1b, 21d970d8de, 280b83fcf3, 2b1dc6d6a5.

Discussion of held-out errors.

3.3 Bonus: direct point-map prediction

Question. The baseline gets 3D points indirectly, by unprojecting predicted depth with predicted cameras. (1) Is a direct world-space point-map head a better way to get 3D points? (2) Does adding point-map supervision as an extra task change the camera and depth predictions?

Method.

Settings (configs/vggt_points.json, run outputs/vggt_points_b4w2_2):

Validation camera/depth loss for both runs, and the point loss of the point model
Point-model training curves (total, camera, depth)

(1) Point head vs. camera + depth unprojection. All rows are scored against the GT canonical points over every GT-valid pixel, in normalized units (scene centroid at distance 1). The script is compare_pointmaps.py; raw results are in pointmap_val.json and pointmap_test.json.

Split Model / point source Mean L2 ↓ Median L2 ↓ Rel. L2 ↓ Acc@0.05 ↑ Acc@0.10 ↑
Val Baseline / camera + depth 0.257 0.159 0.224 0.112 0.296
Val Point model / camera + depth 0.218 0.134 0.196 0.130 0.357
Val Point model / point head 0.235 0.144 0.214 0.102 0.327
Test Baseline / camera + depth 0.240 0.163 0.207 0.103 0.280
Test Point model / camera + depth 0.210 0.136 0.182 0.117 0.351
Test Point model / point head 0.219 0.143 0.190 0.088 0.323

(2) Effect on camera and depth metrics (standard evaluation; baseline numbers are from Q3.2):

Split Model AbsRel ↓ δ1 ↑ Rot AUC@5/15/30 ↑ Trans AUC@5/15/30 ↑ Pose AUC@5/15/30 ↑ Rot / trans err (°) ↓
Val Baseline 0.145 0.832 0.214 / 0.539 / 0.717 0.010 / 0.078 / 0.191 0.004 / 0.060 / 0.173 11.8 / 41.2
Val Point model 0.133 0.843 0.274 / 0.597 / 0.763 0.026 / 0.101 / 0.223 0.011 / 0.081 / 0.206 8.9 / 38.5
Test Baseline 0.159 0.795 0.188 / 0.501 / 0.698 0.013 / 0.076 / 0.194 0.001 / 0.047 / 0.167 10.5 / 39.1
Test Point model 0.154 0.816 0.250 / 0.583 / 0.755 0.010 / 0.067 / 0.190 0.003 / 0.052 / 0.176 9.1 / 38.4

Qualitative comparison on two validation scenes (left to right: ground truth, point model camera + depth, point model point head):

Ground truth Camera + depth Point head

Conclusions.