16-825 Assignment 4: Multi-View Transformers

Liza Dahiya · ldahiya@andrew.cmu.edu

1. Multi-View Data Preprocessing

All Q1 functions are implemented in a4data/geometry.py. The figures use the same five training samples as scripts/visualize_data.py (seed 825, 8 views, 224×224). The red frustum is camera 0 and blue frustums are the other views. Points are colored by RGB and subsampled for display.

1.1–1.3 Unprojection and canonicalization

After unprojection, points and cameras are in the original ScanNet++ world frame, in meters. Camera 0 sits at an arbitrary position, and the centroid of the valid points is several meters from the origin. After canonicalization, camera 0 is the identity pose at the origin, and the valid-point centroid is exactly 1 unit away.

SampleCamera 0 center, world (m)Centroid distance (m) Camera 0 center, canonicalCentroid distance, canonical
036bce3393_frame_004470(3.346, 4.677, 1.468)6.668(0.000, 0.000, 0.000)1.0000
076c822ecc_frame_002890(1.540, 2.767, 1.586)4.435(0.000, 0.000, 0.000)1.0000
079a326597_frame_007430(3.756, 2.082, 1.532)2.718(0.000, 0.000, 0.000)1.0000
07f5b601ee_frame_003630(3.094, 3.489, 1.408)5.118(0.000, 0.000, 0.000)1.0000
07ff1c45bb_frame_000460(1.340, 2.171, 1.516)3.285(0.000, 0.000, 0.000)1.0000

The metric centroid distance (third column) is the scene_scale that divides all translations, depths and points.

Interactive viewer screenshots

Screenshots of outputs/unprojection/index.html (left) and outputs/canonicalization/index.html (right) for the same five samples. The side-panel note under the depth map gives the first camera center and the valid-point centroid distance. After canonicalization these read (0.0000, 0.0000, 0.0000) and 1.000 normalized units for every scene.

036bce3393_frame_004470

Unprojection viewer, 036bce3393_frame_004470
After Q1.2: original world frame (m)
Canonicalization viewer, 036bce3393_frame_004470
After Q1.3: camera-0 frame, normalized units

076c822ecc_frame_002890

Unprojection viewer, 076c822ecc_frame_002890
After Q1.2: original world frame (m)
Canonicalization viewer, 076c822ecc_frame_002890
After Q1.3: camera-0 frame, normalized units

079a326597_frame_007430

Unprojection viewer, 079a326597_frame_007430
After Q1.2: original world frame (m)
Canonicalization viewer, 079a326597_frame_007430
After Q1.3: camera-0 frame, normalized units

07f5b601ee_frame_003630

Unprojection viewer, 07f5b601ee_frame_003630
After Q1.2: original world frame (m)
Canonicalization viewer, 07f5b601ee_frame_003630
After Q1.3: camera-0 frame, normalized units

07ff1c45bb_frame_000460

Unprojection viewer, 07ff1c45bb_frame_000460
After Q1.2: original world frame (m)
Canonicalization viewer, 07ff1c45bb_frame_000460
After Q1.3: camera-0 frame, normalized units
Additional static renders and RGB/depth strips for the same samples

036bce3393_frame_004470

Unprojection, scene 0
After Q1.2: original world frame (meters)
Canonicalization, scene 0
After Q1.3: camera-0 frame, normalized scale
RGB and depth, scene 0
Sampled views (top: RGB; bottom: depth, white = invalid)

076c822ecc_frame_002890

Unprojection, scene 1
After Q1.2: original world frame (meters)
Canonicalization, scene 1
After Q1.3: camera-0 frame, normalized scale
RGB and depth, scene 1
Sampled views (top: RGB; bottom: depth, white = invalid)

079a326597_frame_007430

Unprojection, scene 2
After Q1.2: original world frame (meters)
Canonicalization, scene 2
After Q1.3: camera-0 frame, normalized scale
RGB and depth, scene 2
Sampled views (top: RGB; bottom: depth, white = invalid)

07f5b601ee_frame_003630

Unprojection, scene 3
After Q1.2: original world frame (meters)
Canonicalization, scene 3
After Q1.3: camera-0 frame, normalized scale
RGB and depth, scene 3
Sampled views (top: RGB; bottom: depth, white = invalid)

07ff1c45bb_frame_000460

Unprojection, scene 4
After Q1.2: original world frame (meters)
Canonicalization, scene 4
After Q1.3: camera-0 frame, normalized scale
RGB and depth, scene 4
Sampled views (top: RGB; bottom: depth, white = invalid)

1.7 Final check: check_data.py

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
SampleViewsRes.Valid depthMax point reproj. err. (px) Max ray reproj. err. (px)Centroid distance
e050c15a8d_frame_00708082240.9781.10e-041.08e-041.000000
5f99900f09_frame_00282082240.9901.23e-041.11e-041.000000
47b37eb6f9_frame_00272082240.9619.88e-058.90e-051.000000
4c5c60fa76_frame_00669082240.7521.13e-041.02e-041.000000

Every reprojection error is about 1e-4 px, well under the 0.01 px requirement, and every centroid distance is 1.

Full output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011280165199423209,
      "max_ray_reprojection_error_px": 0.00010225349979009479,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. Absolute ScanNet++ coordinates are arbitrary: each scan has its own origin, orientation and size. To give the network a consistent input distribution, each sample is expressed relative to its first sampled camera. With camera 0's pose $T_{w\leftarrow c_0}$, every pose becomes $T'_i = T_{w\leftarrow c_0}^{-1}\,T_{w\leftarrow c_i}$, computed in float64, so camera 0 is the identity. Depth is unprojected in this frame. The scene scale $s = \lVert \operatorname{mean}(p_\text{valid}) \rVert$ is the distance from camera 0 to the centroid of the valid points pooled over all views. Translations, depths and points are divided by $s$. Rotations, and therefore ray directions, are unchanged. The centroid is not subtracted, so camera 0 stays at the origin. This removes the global rigid transform and scale, which cannot be recovered from images alone.

World-space ray directions. Each pixel center $(u,v)$ is lifted to the camera-frame direction $K^{-1}[u,v,1]^\top$. This vector has unit z, so multiplying it by the z-depth gives the 3D point. Rotating it by the camera-to-world rotation and normalizing gives $d = R\,K^{-1}[u,v,1]^\top / \lVert R\,K^{-1}[u,v,1]^\top \rVert$. Every ray of a view starts at that camera's center $o$, the translation of its pose. The rays therefore encode both the intrinsics (how directions spread across pixels) and the extrinsics (the rotation and the center).

Plücker moment. Each ray is stored as $[d, m]$ with moment $m = o \times d$. The moment is perpendicular to the plane that contains the ray and the origin, and $\lVert m \rVert$ is the ray's distance from the origin. Because $(o + t d)\times d = o\times d$, $m$ does not depend on which point of the ray is used as its origin, so $[d,m]$ identifies the line itself. This gives a dense, per-pixel camera encoding that a transformer can take as image channels. For camera 0, $o = 0$, so its moments are zero.

2. Novel-View Synthesis with Mini-LVSM

Implemented in models/lvsm/model.py (patchify/unpatchify, build_tokens, forward), models/lvsm/losses.py (MSE) and models/lvsm/train.py (bfloat16 autocast forward, float32 loss). Each target view gets its own sequence: the tokens of all source views (RGB in [−1,1] plus Plücker rays), followed by that target's ray-only query tokens. The model never sees target RGB.

2.1 Overfitting

After 2000 steps on one fixed batch (S=2, 6 targets per sample), the model reaches PSNR 31.00 dB (SSIM 0.872) on that batch.

Overfit batch, val_epoch000.png. Columns: source 1, source 2, ground truth, prediction.
Overfit batch, val_epoch000.png. Columns: source 1, source 2, ground truth, prediction.

2.2 Training and evaluation

Training settings

Training curves

Train/validation MSE and PSNR, and validation SSIM, per epoch.
Train/validation MSE and PSNR, and validation SSIM, per epoch.

Validation PSNR rises from 12.2 dB to 16.13 dB and is still climbing at epoch 99. Training PSNR (15.6 dB) is no higher than validation PSNR, so the model is under-fitting, not over-fitting. Its quality is limited by the short training budget (5.6k steps). The overfit experiment shows that the implementation itself can reach >30 dB.

Validation metrics per ablation

python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none. Validation split, 28 scenes, the first 32 prepared frames, 8 views per sample.

Source viewsTarget images Correct raysShuffled raysNo rays (zeros)
PSNR ↑SSIM ↑PSNR ↑SSIM ↑PSNR ↑SSIM ↑
119615.190.55811.800.48812.470.528
216816.130.56011.480.47212.440.527
411216.870.57511.590.47312.450.537

Visualizations (columns: source views, ground truth, prediction)

Training-time validation snapshots at the first and last visualized epochs:

Epoch 0
Epoch 0
Epoch 95
Epoch 95

Evaluation with correct target rays and 1, 2 and 4 source views:

S=1, correct rays
S=1, correct rays
S=2, correct rays
S=2, correct rays
S=4, correct rays
S=4, correct rays

Ray-conditioning ablation with S=2:

S=2, shuffled rays (another target's camera)
S=2, shuffled rays (another target's camera)
S=2, no rays (all zeros)
S=2, no rays (all zeros)

Discussion

Number of source views. With correct rays, PSNR rises from 15.19 dB (S=1) to 16.13 dB (S=2) and 16.87 dB (S=4), and SSIM improves too. This happens even though the model was trained only with S=2. More sources cover more of the scene, so more target pixels are visible in some source, and the model needs to hallucinate less. Full attention accepts any number of source tokens, so the same weights run at S=1 and S=4 without retraining. Going from 2 to 4 views gains less than going from 1 to 2, probably because views sampled from the same 32-frame window overlap a lot, so extra views add less new content.

Ray conditioning. The target Plücker rays are the only signal telling the model which view to render, and removing or corrupting them causes the largest drop of any setting. With S=2, shuffled rays drop PSNR to 11.48 dB (−4.6 dB) and zero rays to 12.44 dB (−3.7 dB), regardless of the number of sources. Shuffled rays are worse than no rays. A likely explanation is that wrong rays make the model confidently render the wrong viewpoint, while zero rays are out of distribution and push it toward a blurry, scene-average image, which costs less under MSE. So the model is using the target camera, not just copying or averaging the sources.

Limitations. Predictions get the coarse layout and colors right but are blurry and show 8×8 patch artifacts. MSE alone rewards predicting the average of plausible images, which looks blurry, and 5.6k steps is a short schedule. I would expect longer training (validation PSNR had not plateaued) and an added perceptual or SSIM loss to help most, but I did not test either.

2.3 Bonus: training with a random number of source views

Question. The baseline always trains with S=2 source views, yet it is evaluated with 1, 2 and 4. Does training with a random number of source views per step (S uniform in {1,2,3,4}, targets = the remaining views) give a model that generalizes better across source counts, especially to S=1 and S=4, which the baseline never sees during training?

Setup (controlled). Identical to the 2.2 baseline (same model, optimizer, LR schedule, 100 epochs, batch size 4, data seed 825, 8 views per sample); the only change is that each training step samples S∈{1,..,4} (new optional train_source_views setting in models/lvsm/train.py; config configs/lvsm_randS.json). Validation during training and checkpoint selection still use S=2, so both runs are selected the same way. Both are evaluated with the same 9-setting ablation as 2.2. Training took about 89 min on a shared A100.

Left: validation PSNR (S=2) during training for both runs. Right: final PSNR vs. number of source views at evaluation (correct rays).
Left: validation PSNR (S=2) during training for both runs. Right: final PSNR vs. number of source views at evaluation (correct rays).
Eval SRaysFixed S=2 (baseline) Random S∈{1..4}Δ PSNR
PSNR ↑SSIM ↑PSNR ↑SSIM ↑
1correct15.190.55815.280.559+0.08
1shuffled11.800.48811.770.486-0.04
1none12.470.52812.500.526+0.03
2correct16.130.56016.230.560+0.10
2shuffled11.480.47211.440.468-0.05
2none12.440.52712.470.522+0.03
4correct16.870.57517.100.575+0.23
4shuffled11.590.47311.470.465-0.12
4none12.450.53712.500.530+0.04
Random-S model, S=1, correct rays (source, GT, prediction).
Random-S model, S=1, correct rays (source, GT, prediction).
Random-S model, S=4, correct rays (sources, GT, prediction).
Random-S model, S=4, correct rays (sources, GT, prediction).

Results. With correct rays, the random-S model is better at every source count: +0.08 dB at S=1, +0.10 dB at S=2 and +0.23 dB at S=4. SSIM is essentially unchanged (within ±0.001). The largest gain is at S=4, which is the setting furthest from what the baseline trained on; the random-S model saw 4-source sequences during training and the baseline never did. At S=2 the random-S model is no worse even though only about a quarter of its training steps used S=2. The shuffled and zero-ray settings stay about equally bad for both models, so the random-S model is just as dependent on correct target rays.

Conclusion and limitations. Random-S training did not hurt at S=2 and helped a little at S=4, so it is a cheap change worth keeping. The gains are small, though (at most 0.23 dB), and come from a single training run per configuration. The epoch-to-epoch fluctuation of the validation curve is about 0.1 dB, so the S=1 and S=2 differences are within plausible run-to-run noise; only the S=4 gain stands somewhat above it, and repeated seeds would be needed to confirm it. Both models remain under-trained (blurry predictions, PSNR still rising at epoch 100), and a longer schedule would likely matter more than the view-count curriculum.

3. Multi-View Geometry with Mini-VGGT

Implemented in models/vggt/model.py (alternating frame/global attention; the last frame-block and last global-block outputs are concatenated to 2D and layer-normed), models/vggt/heads.py (camera MLP with ReLU on FoV; depth = exp of patch-decoded logits plus an RGB-conditioned conv residual; confidence = 1 + exp) and models/vggt/losses.py (translation/quaternion/FoV L1; trimmed regression, confidence-weighted and 4-scale gradient depth losses).

The 3D figures below are static renders I made from the same predictions that the provided HTML viewer shows (submission/make_vggt_figures.py). The red frustum is camera 0. Predicted points are unprojected from the predicted depth using the predicted cameras. GT and prediction share axis limits.

3.1 Overfitting

One fixed training clip, 40 epochs × 25 steps, LR 0.001. On that clip, the final model reaches depth AbsRel 0.037 and δ1 0.955, with a mean rotation error of 0.27° and a mean translation-direction error of 0.32° (pose AUC@30 0.999). The depth loss goes negative, as expected, because of the −0.2 log(confidence) term.

Overfit loss curves (total, camera, depth).
Overfit loss curves (total, camera, depth).

Viewer screenshots (outputs/vggt_overfit/visualization/index.html), same camera angle, switching the geometry selector between ground truth and predicted cameras + depth:

036bce3393_frame_004470 (overfit clip)

Ground truth, 036bce3393_frame_004470 (overfit clip)
Ground truth
Prediction, 036bce3393_frame_004470 (overfit clip)
Predicted camera + depth
Static render of the same clip with GT and predicted depth maps for all 8 views.
Static render of the same clip with GT and predicted depth maps for all 8 views.

3.2 Training and evaluation

Training settings

Training curves

Normal training: total, camera and depth loss (train and validation).
Normal training: total, camera and depth loss (train and validation).

Depth loss converges in the first ~100 epochs. Camera loss keeps decreasing for the whole run, so pose accuracy would likely improve further with more training. Validation loss stays below training loss for the whole run. The two splits are sampled differently (in training, the reference view is a random frame of a random 32-frame window; in validation it is always frame 0 of the prepared clip), and the 28 validation scenes may simply be easier. I did not isolate which of these causes the gap.

Validation and test metrics

python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split {val,test} --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825. 28 scenes and 784 view pairs per split, with no degenerate pairs.

SplitAbsRel ↓δ1 ↑Rotation AUC ↑Translation AUC ↑Joint pose AUC ↑Mean rot. err.Mean transl. err.
@5°@15°@30°@5°@15°@30°@5°@15°@30°
Validation0.1450.8300.2030.5220.6950.0150.0840.1930.0070.0600.16813.2°41.0°
Test0.1690.7900.1990.5220.7170.0150.0770.1850.0030.0550.1649.7°41.1°

GT vs. predicted geometry (four validation scenes)

Viewer screenshots (outputs/vggt_train/visualization/index.html); left ground truth, right prediction from best.pt. The side-panel note under each prediction gives its valid-point centroid distance, which differs from 1 by the model's scale error.

09c1414f1b_frame_000000

Ground truth, 09c1414f1b_frame_000000
Ground truth
Prediction, 09c1414f1b_frame_000000
Predicted camera + depth

21d970d8de_frame_000000

Ground truth, 21d970d8de_frame_000000
Ground truth
Prediction, 21d970d8de_frame_000000
Predicted camera + depth

280b83fcf3_frame_000000

Ground truth, 280b83fcf3_frame_000000
Ground truth
Prediction, 280b83fcf3_frame_000000
Predicted camera + depth

2b1dc6d6a5_frame_000000

Ground truth, 2b1dc6d6a5_frame_000000
Ground truth
Prediction, 2b1dc6d6a5_frame_000000
Predicted camera + depth

Static renders of the same four scenes, with GT and predicted depth maps for all 8 views:

Validation scene 1: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 1: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 2: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 2: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 3: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 3: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 4: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.
Validation scene 4: GT (left) vs. prediction (right), with GT and predicted depth for all 8 views.

Discussion of errors on held-out scenes

Depth generalizes reasonably: AbsRel is 0.145 on validation and 0.169 on test, and δ1 is 0.83 and 0.79. The coarse scene layout, including floor, walls and large furniture, is correct and consistent across views, so the fused point clouds have the right overall shape. Errors concentrate in the details. Predicted depth is smooth and blocky at the 14-px DINO patch scale, and thin structures such as chair legs disappear; the two-layer 3×3 conv refinement on RGB is too small to restore them. Object boundaries are smeared, so the predicted clouds are noisier near depth discontinuities. Windows and other regions with no valid GT depth receive no supervision.

Rotation is learned moderately well: AUC@30 is 0.70 on validation and 0.72 on test, with a mean error of about 13° and 10°. Fine accuracy is still poor, however (AUC@5 is about 0.20).

Translation direction is the main failure mode: AUC@30 is only 0.19 and 0.19, with mean errors of about 41°, and it caps the joint pose AUC. Part of the reason is short baselines: on the validation clips, the median distance between two cameras is 0.29 m against a median scene depth of 2.05 m, and 21% of view pairs are less than 10 cm apart. For such pairs, a translation error of a few centimeters already changes the direction by tens of degrees. The camera loss was still falling when training ended, so longer training may help; VGGT's iterative camera refinement is another option I did not try. Test depth is slightly worse than validation while test rotation is slightly better, which is consistent with scene-to-scene variation across only 28 scenes per split.

3.3 Bonus: how does the number of input views affect Mini-VGGT?

Question. The model was trained with exactly 8 views. There is no view-order embedding (only camera 0 gets its own camera and register tokens), so the same weights accept any number of views. How do accuracy, runtime and memory change when evaluating with 2, 4, 8 or 16 views, and is "more views is better" true for this model?

Setup. The trained best.pt from 3.2, with no retraining, evaluated on the validation split using the same protocol as the official evaluation (first 32 prepared frames, 224px, seed 825); only num_views changes. The loader always keeps frame 0 as the reference and draws the other V−1 views at random from frames 1–31. The V=8 row reproduces the official validation numbers exactly. Latency (median of 10 forward passes after warm-up) and peak activation memory (excluding the weights) are measured for one sample, in bf16, on an A100. Script: submission/bonus_vggt_views.py. To separate scale errors from shape errors, I also report AbsRel after per-sample median scale alignment (submission/bonus_vggt_views_scale.py).

ViewsPose pairsAbsRel ↓AbsRel, scale-aligned ↓ |log scale err.| ↓δ1 ↑Rot. AUC@30 ↑Transl. AUC@30 ↑ Pose AUC@30 ↑Mean rot. err.Peak mem. (MiB)Latency (ms)
2280.1090.0800.0640.8790.6980.2360.20113.7°14371
41680.1330.0870.0890.8450.6710.1740.14813.3°16079
87840.1450.0900.0980.8300.6950.1930.16813.2°202106
1633600.1700.0920.1080.8070.6730.1680.14513.4°27996
Depth error, pose AUC@30, latency and memory vs. number of input views (dotted line: training view count).
Depth error, pose AUC@30, latency and memory vs. number of input views (dotted line: training view count).

Results.

Conclusion and limitations. For this small model, adding views does not improve per-view geometry. It mostly makes the scene-normalized scale harder to predict, while relative depth shape and pose stay nearly constant. Depth metrics computed after scale alignment would separate the two effects, and training with a variable view count might let the model benefit from extra views; I did not test the latter for VGGT. Limitations: one checkpoint, one split (28 scenes) and one random view draw per V; small-V pose metrics rest on few pairs; and latency could not be measured cleanly on the shared GPU.