16-825 Assignment 4: Multi-View Transformers

All runs use one NVIDIA RTX 4070 Ti SUPER (16 GB) and the ScanNet++ iPhone subset (224 training, 28 validation and 28 test scenes).

1. Multi-View Data Preprocessing

Unprojection (Q1.1–1.2)

Points and cameras in the original ScanNet++ world frame, in meters (5 training scenes, 10,000 points each).

Unprojection, scene 1 (original world frame, meters)
Unprojection, scene 1 (original world frame, meters)
Unprojection, scene 2 (original world frame, meters)
Unprojection, scene 2 (original world frame, meters)
Unprojection, scene 3 (original world frame, meters)
Unprojection, scene 3 (original world frame, meters)
Unprojection, scene 4 (original world frame, meters)
Unprojection, scene 4 (original world frame, meters)
Unprojection, scene 5 (original world frame, meters)
Unprojection, scene 5 (original world frame, meters)

Canonicalization (Q1.3)

The same frames after canonicalization. The side panel shows that the first camera center is (0, 0, 0) and that the valid-point centroid distance is 1.000 normalized units.

Canonicalization, scene 1 (first-camera frame, normalized units)
Canonicalization, scene 1 (first-camera frame, normalized units)
Canonicalization, scene 2 (first-camera frame, normalized units)
Canonicalization, scene 2 (first-camera frame, normalized units)
Canonicalization, scene 3 (first-camera frame, normalized units)
Canonicalization, scene 3 (first-camera frame, normalized units)
Canonicalization, scene 4 (first-camera frame, normalized units)
Canonicalization, scene 4 (first-camera frame, normalized units)
Canonicalization, scene 5 (first-camera frame, normalized units)
Canonicalization, scene 5 (first-camera frame, normalized units)

Final check (Q1.7)

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

SampleViewsResolutionValid depth fractionMax point reprojection error (px)Max ray reprojection error (px)Centroid distance
e050c15a8d_frame_00708082240.97791.10e-049.41e-051.0000
5f99900f09_frame_00282082240.99031.11e-041.11e-041.0000
47b37eb6f9_frame_00272082240.96109.35e-058.90e-051.0000
4c5c60fa76_frame_00669082240.75171.07e-049.42e-051.0000

Batch shapes (batch size 2):

rgb(2, 8, 3, 224, 224)
depth(2, 8, 1, 224, 224)
depth_valid(2, 8, 1, 224, 224)
K(2, 8, 3, 3)
frame_numbers(2, 8)
T_world_from_camera(2, 8, 4, 4)
original_world_from_reference(2, 4, 4)
scene_scale(2)
points_world(2, 8, 3, 224, 224)
ray_origins(2, 8, 3, 224, 224)
ray_directions(2, 8, 3, 224, 224)
plucker(2, 8, 6, 224, 224)
Full check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.351845073979348e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010746094631031156,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

All reprojection errors are about 10-4 px, well below the 0.01 px limit, and every centroid distance is 1.

Explanations

Canonicalization. A ScanNet++ world frame has an arbitrary origin, orientation and metric scale, so the same local geometry can appear with very different coordinates. We remove this ambiguity per sample. Every pose is expressed relative to the first sampled camera, Tref←cami = (Tworld←ref)-1 Tworld←cami, so camera 0 becomes the identity. Depth is then unprojected into this frame, and the scale s is the distance from the origin to the centroid of all valid points. Translations, depth and points are divided by the same s, so the centroid lies at distance 1. Rotations are unchanged and the centroid is not subtracted, so camera 0 stays exactly at the origin. Every training sample therefore shares the same coordinate and scale conventions, which is what the models are supervised and conditioned on.

World-space ray directions. For pixel (u, v), K-1[u, v, 1]T is the camera-space direction through that pixel center; its z component is 1. The camera-to-world rotation R takes it to world space, d = R K-1[u, v, 1]T / ||·||. The translation does not affect a direction; it only sets the ray origin o, the camera center t. Every pixel of a view shares the same origin, and its direction encodes both the intrinsics (through K) and the camera orientation (through R).

Plücker moment. A ray is stored as (d, m) with m = o × d. The moment does not depend on which point of the line is used: (o + λd) × d = o × d, because d × d = 0. So (d, m) describes the line itself. The moment is perpendicular to the plane containing the line and the world origin, and for a unit d its norm is the distance from the origin to the line. This gives each pixel a smooth, dense 6-D camera encoding that a transformer can consume like an image channel.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit

After 2000 steps on one fixed batch (4 samples, S = 2 sources and 6 targets each), the final PSNR on that batch is 30.92 dB (SSIM 0.870, MSE 0.00087). The loss decreases steadily; the running average over all 2000 steps is 25.39 dB.

val_epoch000.png: source views, ground truth and prediction
val_epoch000.png: source views, ground truth and prediction

2.2 Training and evaluation

Training settings: configs/lvsm.json. The model uses patch size 8, width 384, 8 transformer blocks, head dimension 64, MLP ratio 4 and QK-norm (14.6 M parameters). Each sample has 8 views from a 32-frame window at 224×224, with S = 2 sources and T = 6 targets. Training uses batch size 4 for 100 epochs (56 steps each, 5,600 steps) with AdamW (lr 2×10-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps then cosine decay to 10%, gradient clipping 1.0 and bfloat16 autocast. Validation uses 8 views from the first 32 prepared frames. Training took about 32 minutes. best.pt is from epoch 99, with validation PSNR 16.13 dB and SSIM 0.560.

Mini-LVSM training curves

Training visualizations (columns: source views, ground truth, prediction):

Validation batch, epoch 0
Validation batch, epoch 0
Validation batch, epoch 50
Validation batch, epoch 50
Validation batch, epoch 95
Validation batch, epoch 95

python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none

Source viewsCorrect raysShuffled raysNo raysTarget images
PSNRSSIMPSNRSSIMPSNRSSIM
115.190.55711.810.48712.470.530196
216.130.56011.490.47212.440.528168
416.880.57511.590.47312.450.537112

Evaluation visualizations on the validation split (columns: source views, ground truth, prediction):

S = 1, correct target rays
S = 1, correct target rays
S = 1, shuffled target rays
S = 1, shuffled target rays
S = 1, no target rays (zeros)
S = 1, no target rays (zeros)
S = 2, correct target rays
S = 2, correct target rays
S = 2, shuffled target rays
S = 2, shuffled target rays
S = 2, no target rays (zeros)
S = 2, no target rays (zeros)
S = 4, correct target rays
S = 4, correct target rays
S = 4, shuffled target rays
S = 4, shuffled target rays
S = 4, no target rays (zeros)
S = 4, no target rays (zeros)

Number of source views. With correct rays, quality rises steadily with more sources: PSNR goes from 15.19 dB (S = 1) to 16.13 dB (S = 2) to 16.88 dB (S = 4), even though the model was only trained with S = 2. More sources cover more of the scene that the target camera sees and make the geometry less ambiguous, so less content has to be hallucinated. The transformer also handles a variable number of source tokens, because nothing in the model depends on S.

Ray-conditioning ablation. Removing the target rays costs 2.7–4.4 dB, and shuffling them costs 3.4–5.3 dB; the gap grows with S because the correct-ray model improves with more sources. Shuffled rays are consistently worse than no rays. With zero rays the model knows nothing about the target pose and outputs an almost uniform image in the scene's mean color, the MSE-optimal guess when the viewpoint is unknown. With shuffled rays it confidently renders a different, valid viewpoint that is misaligned with the ground truth, which is penalized more. With wrong or missing rays, quality also no longer improves with more source views. So the model really uses the target Plücker rays; it does not simply copy a source image. Predictions are still blurry overall, with patch-grid artifacts, which is expected from a 14.6 M-parameter model trained for 5,600 steps on 224 scenes with a pure MSE loss. The validation PSNR was still rising at epoch 99, so longer training would help.

2.3 Bonus: Training view count

Question. The baseline always trains with S = 2 source views, yet it is evaluated with 1, 2 and 4. Does training with a random number of source views per step make the model better across evaluation view counts, especially at S = 4, which it has never seen during training?

Setup. configs/lvsm_random_views.json is identical to the baseline (same model, data, optimizer, schedule, 100 epochs, seed and validation protocol) except that train_source_views = [1, 2, 3, 4]. At every training step S is drawn uniformly from this set, and the remaining 8 − S views of the clip are targets. Validation during training and model selection still use S = 2, as in the baseline. The compute per step is nearly unchanged: the attention cost scales with T(S+1) and averages 18 over {1, 2, 3, 4}, the same as for S = 2, so the runs took 20.1 s and 19.1 s per epoch. Both best.pt checkpoints (epoch 99 in both runs) are evaluated with correct target rays at S = 1, 2, 4, using python -m models.lvsm.eval --checkpoint outputs/lvsm_random_views/best.pt --num-source-views 1 2 4 --conditioning correct.

Source views at evaluationPSNR (dB)SSIM
Fixed S = 2Random SΔFixed S = 2Random SΔ
115.1915.28+0.090.5570.559+0.002
216.1316.24+0.110.5600.560+0.000
416.8817.11+0.230.5750.575+0.001
PSNR and SSIM versus evaluation source views Training curves for fixed and random source-view training

Validation scenes, correct target rays (columns: source views, ground truth, prediction).

Fixed S = 2 model, S = 1
Fixed S = 2 model, S = 1
Fixed S = 2 model, S = 2
Fixed S = 2 model, S = 2
Fixed S = 2 model, S = 4
Fixed S = 2 model, S = 4
Random-S model, S = 1
Random-S model, S = 1
Random-S model, S = 2
Random-S model, S = 2
Random-S model, S = 4
Random-S model, S = 4

Results. Random-S training is better at every evaluation view count: +0.09 dB at S = 1, +0.11 dB at S = 2 and +0.23 dB at S = 4, with SSIM equal or slightly higher. The gain is largest at S = 4, the setting the baseline never trained on, and smallest at S = 1. The training curves show the random-S model slightly ahead for most of training (for example 15.65 vs 15.59 dB at epoch 50 and 16.24 vs 16.13 dB at epoch 99, both measured at S = 2). It also loses nothing at S = 2, the baseline's own training setting.

Interpretation. With a fixed S = 2, every sequence has exactly 2×N source tokens. The model learns to aggregate information from two views, and at test time it must extrapolate to a longer context. Randomizing S exposes it to 1–4 sources, so it learns to use additional views when available and to fall back on a single view when not. This acts as data augmentation over context length, at no extra cost. Visually the two models are nearly indistinguishable; both are still blurry and show the same patch artifacts, so the improvement is a small overall error reduction rather than new detail.

Limitations. The improvements are small (0.1–0.2 dB) and come from a single seed. Run-to-run variation of this size is plausible, so the consistent sign across all three view counts and across most of training is more convincing than any single number. A stronger test would use several seeds, train longer (both models were still improving at epoch 99), include S > 4, and select the checkpoint with a mix of view counts instead of S = 2 only.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit (one fixed training clip, 40 epochs × 25 steps, lr 10-3).

Overfit loss curve: total, camera and depth loss
Overfit loss curve: total, camera and depth loss

On the fixed clip after 40 epochs: depth AbsRel 0.037, δ1 0.954, rotation error 0.18°, translation-direction error 0.42°, and joint pose AUC@5/15/30 of 0.971/0.990/0.995. The depth loss becomes negative because of the −0.2 log(confidence) term, which is expected once the confidence is large.

Overfit clip: ground truth
Overfit clip: ground truth
Overfit clip: prediction
Overfit clip: prediction

3.2 Training and evaluation

Training settings: configs/vggt.json, unchanged. The encoder is a frozen DINOv2 ViT-B/14 followed by 8 frame/global attention pairs (width 384, 6 heads), a 4-block camera head and a depth head with RGB refinement. Training uses 8 views from a 32-frame window at 224×224, batch size 16, AdamW (lr 6×10-4, weight decay 0.01), 10 warmup epochs then cosine decay to 5%, gradient clipping 1.0, loss weights camera 5 and depth 1, and bfloat16 attention. The run lasted 1000 epochs (14 steps each, 14,000 steps) and took 3 h 38 min on one RTX 4070 Ti SUPER, using about 15 GB of GPU memory. best.pt is from epoch 843, selected by validation loss.

Normal training loss curve: total, camera and depth loss
Normal training loss curve: total, camera and depth loss

Evaluation of best.pt: 8 views from the first 32 prepared frames, 224px, seed 825, full splits.

MetricValidationTest
Depth AbsRel ↓0.1470.167
Depth δ1 ↑0.8100.783
Rotation AUC@5° ↑0.2020.198
Rotation AUC@15° ↑0.5410.524
Rotation AUC@30° ↑0.7080.723
Translation AUC@5° ↑0.0170.014
Translation AUC@15° ↑0.0850.067
Translation AUC@30° ↑0.1940.177
Joint pose AUC@5° ↑0.0070.004
Joint pose AUC@15° ↑0.0640.048
Joint pose AUC@30° ↑0.1700.162

GT and predicted geometry for four different validation scenes (each prediction uses the predicted cameras and the predicted depth):

Validation scene 1: ground truth
Validation scene 1: ground truth
Validation scene 1: prediction
Validation scene 1: prediction
Validation scene 2: ground truth
Validation scene 2: ground truth
Validation scene 2: prediction
Validation scene 2: prediction
Validation scene 3: ground truth
Validation scene 3: ground truth
Validation scene 3: prediction
Validation scene 3: prediction
Validation scene 4: ground truth
Validation scene 4: ground truth
Validation scene 4: prediction
Validation scene 4: prediction

Errors on held-out scenes. Depth transfers well to unseen scenes. About 80% of the pixels are within 25% of the ground truth (δ1 0.810 on validation, 0.783 on test), and the reconstructed room layouts (floors, walls, tables) are recognizable. The validation and test numbers are close, so the model is not tuned to the validation scenes. The predicted depth is smoother than the ground truth: thin structures such as table legs and chair frames blur into the surfaces behind them, and far regions are smeared, which makes the predicted clouds look puffier. Rotation is moderately accurate, with AUC@30 of 0.708 on validation. Translation direction is the weak point (AUC@30 0.194), and it caps the joint pose AUC (0.170). Over the 7 non-reference cameras of every validation clip, the predicted camera-to-reference distances have about the right size: the median predicted/GT ratio is 1.03. Large displacements are underestimated, though (mean distance 0.17 predicted vs 0.24 GT), and the direction is poor (mean cosine 0.55 between predicted and GT camera offsets). The clips span only 32 prepared frames, so the median GT camera offset is just 0.14 normalized units, about 14% of the distance to the scene centroid. At such small baselines a small position error rotates the translation direction a lot, so this metric is much harder than rotation. The camera loss was still decreasing at epoch 1000, so longer training (the reference used a larger GPU budget), unfreezing DINO or adding a point-map head would likely improve poses. The validation loss sits below the training loss for the whole run. The training loss is averaged over randomly sampled clips while the weights change, whereas validation is a fixed set of 28 clips, one per scene.

3.3 Bonus: Number of views

Question. How does the number of input views affect Mini-VGGT's depth and pose accuracy, runtime and memory? Does a model trained with fewer views generalize to more views at test time, and does training with more views help?

Setup. Two models are compared: the Q3.2 baseline trained with V = 8 views, and a second model trained with V = 4 views (configs/vggt_4views.json). The second model is identical in every other respect: same architecture, frozen DINOv2 ViT-B/14, optimizer, schedule, 1000 epochs, batch size and seed. Both best.pt checkpoints are evaluated on the 28 validation scenes with V ∈ {2, 4, 8, 16, 24, 32}, using scripts/vggt_view_sweep.py. As in the provided evaluation loader, each sample keeps frame 0 as the reference and draws the other V − 1 views with a fixed seed from the same 32-frame window (eval_num_frames = 32, seed 825). Larger V therefore means denser sampling of the same part of the scene, not wider coverage. Runtime is the mean batch-1 forward time over 10 samples after one warm-up, including the frozen encoder. Memory is the peak allocated by PyTorch. Both are measured on an RTX 4070 Ti SUPER.

Accuracy, runtime and memory versus number of views
Trained viewsEval viewsAbsRel ↓δ1 ↑Rot AUC@30 ↑Trans AUC@30 ↑Pose AUC@30 ↑ms / samplePeak GPU memory (GB)
820.1180.8540.6960.2200.19211.50.69
840.1300.8480.7100.2080.17714.60.71
880.1470.8100.7080.1940.17021.30.75
8160.1710.7920.6630.1710.15034.90.82
8240.1690.7830.6640.1720.15152.50.88
8320.1690.7790.6560.1690.14769.90.95
420.1060.8860.6930.2360.20511.60.69
440.1330.8540.6590.1850.15914.50.71
480.1480.8140.6790.1880.15821.20.74
4160.1740.7830.6620.1520.13035.20.82
4240.1770.7600.6690.1460.12752.20.88
4320.1800.7540.6570.1390.12170.10.95
Loss curve of the model trained with 4 views
Loss curve of the model trained with 4 views

Results.

Interpretation. The camera token of frame 0 is the only view-specific marker, and global attention has no view-order embedding, so the architecture accepts any V. What changes with V is how many tokens each global-attention softmax spreads over. A model trained on 4×261 tokens sees 8× longer sequences at V = 32. Its attention statistics fall outside the training distribution, and per-view depth gets noisier, which matches the steeper decline of the 4-view model. The 8-view model also sees 28 camera pairs per sample instead of 6, which is far more relative-pose supervision per step. This plausibly explains why it estimates rotation better even at V = 4. Conversely, the 4-view model's better depth at V = 2 suggests that it specializes in short contexts, where each view's tokens make up a larger share of the global context. The falling translation AUC is partly a property of the metric. With more views from the same 32-frame window, more pairs have very small baselines, whose translation direction is poorly defined; the joint pose AUC inherits this.

Limitations.