Assignment 4: Multi-View Transformers

16-825: Learning for 3D Vision · Kashu Yamazaki

1. Multi-View Data Preprocessing

1.1–1.5 — Implementation

Unprojection (original world frame, metres)

Points from all 8 views overlap into one consistent room, so the depth unprojection and the camera-to-world poses agree.

Unprojected points and cameras, scene 1
036bce3393, frame 4470
Unprojected points and cameras, scene 2
076c822ecc, frame 2890
Unprojected points and cameras, scene 3
079a326597, frame 7430

Canonicalization (first-camera frame, normalized units)

The text panel on the right of each screenshot shows the first camera centre (0, 0, 0) and a valid-point centroid distance of 1.000.

Canonicalized geometry, scene 1
036bce3393, frame 4470
Canonicalized geometry, scene 2
076c822ecc, frame 2890
Canonicalized geometry, scene 3
079a326597, frame 7430

1.7 — check_data.py

SampleValid depthMax reproj. err (px)Max ray reproj. err (px)Centroid dist.
e050c15a8d_frame_0070800.9781.1e-041.1e-041.000
5f99900f09_frame_0028200.9901.2e-041.1e-041.000
47b37eb6f9_frame_0027200.9619.8e-058.9e-051.000
4c5c60fa76_frame_0066900.7521.1e-041.0e-041.000
Full output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011141905270051211,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.777836385183036e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011162471491843462,
      "max_ray_reprojection_error_px": 0.00010225349979009479,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. Absolute ScanNet++ poses live in an arbitrary per-scene world frame and metric scale, which the model cannot observe from images. Re-expressing every pose relative to the first sampled camera (T0−1Ti) removes the arbitrary rotation/translation, and dividing by the distance to the valid-point centroid removes the unobservable global scale. Every sample then has camera 0 at the identity looking down +Z, with scene content about one unit away, so targets have a consistent, learnable distribution. Only one scale factor is used, so relative geometry is preserved, and the centroid is not subtracted, so camera 0 stays at the origin.

World-space ray directions. A pixel's camera-space direction is K−1(u, v, 1). Rotating it by the camera-to-world rotation R (no translation, because directions are vectors) and normalizing gives the unit direction in the canonical frame. All views therefore share one coordinate system, and the rays encode both the intrinsics (ray spread) and the orientation of each camera.

Plücker moment. A ray through origin o with direction d has moment m = o × d. The moment is the same for any point o + λd on the line (since d × d = 0), so [d, m] describes the line itself, not a particular origin. |m| is the distance from the world origin to the line, and the direction of m is the normal of the plane containing the line and the origin. Camera 0's rays all have m = 0, because its centre is the origin. Per-pixel Plücker maps give the transformer a dense, smooth camera encoding.

2. Mini-LVSM: Novel View Synthesis

2.0 — Implementation

2.1 — Overfitting one batch

Final overfit validation PSNR = 30.94 dB (SSIM 0.871, MSE 8.6e-4) after 2000 steps on one fixed batch of 4 sequences (S = 2, T = 6). With the default batch size of 4, the memorization check passes the ≥ 30 dB target. (A first attempt with the larger batch of 8 that I use for full training reached 28.5 dB in the same 2000 steps, because it has twice as many images to memorize.)

Overfit val_epoch000 visualization
val_epoch000.png from the overfit run. Columns: 2 source views, ground truth, prediction.

2.2 — Training settings

SettingValue
Modelpatch 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm (14.6M parameters)
Data224 train scenes, 8 views sampled from a 32-frame window, S = 2 sources / T = 6 targets, 224×224
OptimisationAdamW, lr 2e-4, betas (0.9, 0.95), weight decay 0.05, 500 warmup steps, then cosine decay to 10%; grad clip 1.0
Schedulebatch 8 sequences (48 target images), 400 epochs = 11.2k steps, bf16 autocast
Hardware1× NVIDIA L40S, ~18 s/epoch, about 2 h in total
Selectionbest.pt by validation PSNR (prepared val split, first 32 frames)

Training curves

LVSM training curves
Train and validation MSE (log scale), PSNR and validation SSIM per epoch. There is no overfitting gap. Validation is even slightly higher than train. Training metrics are averaged over random view windows during the epoch, while validation uses a fixed, different set of scenes, so the offset reflects the data, not generalization.

Validation ablations (best.pt)

SourcesCorrect PSNRSSIMShuffled PSNRSSIMNone PSNRSSIM
S = 115.860.58611.710.48312.400.518
S = 217.090.59511.280.46112.420.515
S = 418.200.61411.330.45712.450.522

Test split with true rays: S = 1: 16.25 dB / 0.566, S = 2: 17.21 dB / 0.578, S = 4: 18.47 dB / 0.585.

Training visualizations

LVSM val visualization epoch 0
val_epoch000.png (epoch 0)
LVSM val visualization epoch 380
val_epoch380.png (epoch 380)

Evaluation visualizations (columns: source views, GT, prediction)

S=1 correct
S = 1, correct rays
S=2 correct
S = 2, correct rays
S=4 correct
S = 4, correct rays
S=2 shuffled
S = 2, shuffled rays (another target's rays)
S=2 none
S = 2, no rays (all zeros)
All other ablation images
S=1 shuffled
S = 1, shuffled
S=1 none
S = 1, none
S=4 shuffled
S = 4, shuffled
S=4 none
S = 4, none
test S=2 correct
Test split, S = 2, correct

Discussion

Number of source views. The model was trained only with S = 2, yet quality still increases monotonically with more source views at test time: 15.86 → 17.09 → 18.20 dB PSNR for S = 1 → 2 → 4. The source tokens are simply concatenated, so the transformer can attend to more context for free. With one view, much of the target is unobserved, and the model produces a blurry mean colour there. With four views, most target content has been seen from some source, so the layout and colours are copied more faithfully.

Ray conditioning. The target Plücker rays are the only signal saying where to render. With shuffled rays, PSNR falls by about 5.8 dB (S = 2), even below none. The model confidently renders a plausible image from the wrong viewpoint, and the misaligned image is penalized more heavily than a hedge. With none (zero rays), the model receives an out-of-distribution query and outputs a blurry average of the sources (about 12.4 dB regardless of S). In both ablations, extra source views no longer help, which confirms that the model really uses the camera geometry rather than copying the nearest source. The correct-ray predictions are spatially right but blurry, with visible 8×8 patch seams, as expected for a 14.6M-parameter MSE-trained model after 2 GPU-hours. MSE favours the conditional mean under uncertainty.

2.3 — Bonus: training with a random number of source views

Question. The baseline always trains with S = 2 source views, but it is evaluated with 1, 2 and 4. Does training with a random number of source views per step, S ~ U{1, 2, 3, 4}, make the model better at every evaluation S, and in particular at S = 1 and S = 4, which the baseline never sees?

Setup. I added an optional train_num_source_views list to the config. At each training step, train_one_epoch draws S from it, and the remaining T = 8 − S views are the targets (configs/lvsm_randS.json). Everything else is identical to the baseline: model, data, batch 8, 400 epochs / 11.2k steps, learning-rate schedule and data seed. Per-epoch validation and best.pt selection still use S = 2. Both models are evaluated with the same models.lvsm.eval ablation grid. Since T = 8 − S, the random-S model sees slightly fewer target images per step (5.5 vs 6 on average) but more context per target.

Eval settingFixed S=2 PSNRRandom S PSNRΔFixed SSIMRandom SSIM
S = 1, correct15.8615.92+0.060.5860.590
S = 2, correct17.0917.21+0.120.5950.600
S = 2, shuffled11.2811.23-0.050.4610.460
S = 2, none12.4212.40-0.020.5150.524
S = 4, correct18.2018.42+0.210.6140.622
Fixed vs random S training curves
Training curves, fixed S = 2 vs. random S. Validation always uses S = 2. Training metrics of the random-S run mix all S values.
Fixed S=2 model at S=4
Fixed-S model, evaluated at S = 4
Random S model at S=4
Random-S model, evaluated at S = 4
Fixed S=2 model at S=1
Fixed-S model, evaluated at S = 1
Random S model at S=1
Random-S model, evaluated at S = 1

Results. Random-S training improves every correct-ray setting, and the gain grows with S: +0.06 dB at S = 1, +0.12 dB at S = 2 and +0.21 dB at S = 4 (SSIM +0.004 / +0.005 / +0.007). The S = 2 validation curves overlap for most of training, and the random-S model only pulls slightly ahead in the last ~100 epochs. The ablations with shuffled or no rays stay at the same low level (±0.1 dB), so the extra quality comes from better use of correctly-posed sources, not from a different prior.

Interpretation. The biggest gain is at S = 4, where the baseline is out of distribution: it has never had to attend over four source views, and random-S training makes long-context aggregation part of training. At S = 1, the gain is small even though the model trained with single views 25% of the time. There, the error is dominated by content that is simply not visible in the one source view, and a deterministic MSE model can only predict a blurry mean for it, however it was trained. Training with more sources on average (2.5) may also give cleaner gradients for learning ray-to-pixel correspondence, which also helps S = 2.

Limitations. The differences are small (0.1–0.2 dB) and come from a single seed per setting, so only the trend (gain increasing with S) should be trusted, not the exact numbers. The runs used the same number of steps, but the random-S run ran on a slower GPU (A6000, 4.1 h vs 2.1 h on an L40S). Both models are still improving at 400 epochs, so the gap at convergence may differ. A curriculum, or sampling larger S more often, could be explored next.

3. Mini-VGGT: Cameras and Depth from Images

3.0 — Implementation

3.1 — Single-scene overfit

After 40 epochs (1000 steps) on one fixed scene, the model fits it almost exactly: depth AbsRel 0.036, δ1 0.956, mean rotation error 0.31°, mean translation-direction error 0.29°, rotation AUC@5 1.000, translation AUC@5 0.986, joint pose AUC@5 0.986 (AUC@30 0.998). The total loss goes negative because the −α log c term dominates once the depth error is small.

VGGT overfit loss curve
Overfit loss curve (total, camera, depth).
Overfit GT geometry
Ground truth: cameras and back-projected depth
Overfit predicted geometry
Prediction: decoded cameras and predicted depth

3.2 — Training settings

SettingValue
ModelFrozen DINOv2 ViT-B/14 encoder (224 px, 16×16 patches); aggregator width 384, 6 heads, 8 alternating blocks (4 frame + 4 global)
Data8 views sampled from a 32-frame window, 224×224, 224 training scenes (unchanged defaults)
OptimisationAdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs, then cosine decay to 5%; loss weights camera 5 / depth 1
ScheduleGlobal batch 16 (4 per GPU), 1000 epochs = 14k steps, bf16 autocast
Hardware4× NVIDIA L40 with DDP (torchrun), 4 h 13 min in total (~15 s/epoch including validation)
Selectionbest.pt = lowest validation loss (epoch 896 of 1000)

Loss curve

VGGT training loss curve
Full training: total, camera and depth loss (train and validation).

Validation / test metrics (8 views, first 32 frames, 224 px, seed 825)

SplitAbsRel ↓δ1 ↑Rot@5Rot@15Rot@30
Validation0.13880.8360.1980.5190.704
Test0.15250.7950.1920.5360.735
SplitTrans@5Trans@15Trans@30Pose@5Pose@15Pose@30
Validation0.0180.0880.2010.0050.0690.179
Test0.0060.0650.1850.0020.0480.168
SplitLossCam lossDepth lossMean rot. err (°)Mean trans. err (°)Depth RMSE (norm.)Pose pairs
Validation0.8090.1100.25912.4740.440.191784
Test0.7520.0990.2599.3740.060.197784

GT vs. predicted geometry on four validation scenes

Left: GT cameras and depth; right: predicted cameras and depth, from the provided viewer (visualization/index.html).

09c1414f1b ground truth
09c1414f1b: ground truth
09c1414f1b prediction
09c1414f1b: predicted cameras + depth
21d970d8de ground truth
21d970d8de: ground truth
21d970d8de prediction
21d970d8de: predicted cameras + depth
280b83fcf3 ground truth
280b83fcf3: ground truth
280b83fcf3 prediction
280b83fcf3: predicted cameras + depth
2b1dc6d6a5 ground truth
2b1dc6d6a5: ground truth
2b1dc6d6a5 prediction
2b1dc6d6a5: predicted cameras + depth

Discussion of held-out errors

Depth generalizes reasonably well: AbsRel is 0.139 (val) / 0.153 (test), and 84% / 80% of pixels fall within 1.25× of GT. The screenshots show that the coarse room layout and the global scale are recovered: walls, floor and tables sit in the right place. The predicted depth maps are much smoother than GT, though. Thin structures (chair legs in 09c1414f1b, the chair in 21d970d8de) are averaged into their background. Depth discontinuities are blurred, so the point clouds show "flying" points between foreground and background. Long, low-texture hallways (280b83fcf3) are compressed: the far end is predicted too close. These are the typical failure modes of a dense head on frozen 14-pixel patches, which uses only a light RGB refinement, under an L1/confidence loss that favours the median depth.

Rotations are mostly right: rotation AUC@30 is 0.70 (val) / 0.74 (test), with a mean error of 12.5° / 9.4°. Trajectories like 2b1dc6d6a5 are recovered almost exactly. Fine accuracy is still weak (AUC@5 ≈ 0.2). Translation direction is the main error source: the mean direction error is about 40°, and translation AUC@30 is only 0.20 / 0.18. The joint pose AUC, which takes the worse of the two errors, is therefore limited by translation. Within a 32-frame window, many handheld iPhone baselines are short compared with the scene depth. The direction of such a small translation is ill-conditioned: a tiny absolute position error becomes a large angular error. In 09c1414f1b, one far camera (frame 290) is pulled into the cluster of nearby cameras. The model was also still improving when training ended: the validation loss kept decreasing until epoch 896, and the camera loss had not plateaued. A longer schedule, unfreezing DINO, or more than 224 training scenes should help most.

Val vs. test. The test split has slightly worse depth (AbsRel +0.014) but better rotation. Both splits contain only 28 scenes, so these differences are within scene-to-scene variance. There is no sign of overfitting: the validation loss was still decreasing at the end, so the model is capacity- and schedule-limited.

3.3 — Bonus: how does the number of input views affect accuracy, runtime and memory?

Question. Mini-VGGT uses global attention over the tokens of all views. How do accuracy, runtime and memory depend on the number of input views V, both at evaluation time and during training?

Setup. Two models with the same architecture and hyperparameters (global batch 16, 1000 epochs, same schedule) differ only in the number of training views: the main model (trained V = 8) and a second model (trained V = 4, 2× L40, 2 h 41 min). Each is evaluated on the full validation split with V ∈ {2, 4, 8, 16, 24, 32}, in two regimes (scripts/vggt_views_study.py). The prepared loader always keeps frame 0 as the reference and samples the other V−1 frames from the window with seed 825.

Runtime and peak memory are measured for one sample (B = 1, bf16, including the frozen encoder and the weights) on an L40S, after warm-up. Depth metrics average over all pixels of all V frames, and pose AUCs over all V(V−1) ordered pairs, so V = 2 has only 56 pairs and its pose numbers are noisy.

Accuracy, runtime and memory vs number of views
Validation accuracy vs. number of evaluation views, for both models and both window regimes, plus runtime and memory. Runtime and memory are shown once, because they only depend on V.

Fixed 32-frame window

TrainedEval viewsAbsRelδ1Rot@15Pose@15Pose@30Forward (ms)Peak mem (GB)
V=820.11020.8710.5050.0640.20519.80.74
V=840.12400.8550.5000.0560.16220.10.76
V=880.13880.8360.5190.0690.17920.50.80
V=8160.16590.8060.4730.0460.14420.40.89
V=8240.16940.7900.4730.0470.14829.40.97
V=8320.17550.7820.4720.0480.14938.21.05
V=420.11110.8840.4950.0360.105––
V=440.13130.8560.5270.0450.130––
V=480.14810.8330.4970.0360.126––
V=4160.18250.7790.4350.0250.102––
V=4240.18450.7540.4370.0260.104––
V=4320.19060.7490.4200.0260.099––

Timing and memory are listed only for the V = 8 model (L40S). The V = 4 study ran on a slower L40, and the architecture is identical.

Window = 4V (trained V = 8 model)

Eval viewsFrame windowAbsRelδ1Rot@15Pose@15Pose@30
280.08760.9270.7600.0210.052
4160.11040.8910.6220.0570.136
8320.13880.8360.5190.0690.179
16640.22520.6620.3300.0470.130
24960.34520.5480.2210.0240.090
321280.38940.4380.1850.0170.071

More views at test time do not help this model. In the fixed window, the V = 8 model's depth AbsRel rises from 0.110 (V = 2) to 0.139 (V = 8) and 0.176 (V = 32), while rotation AUC@15 stays roughly flat (0.51 → 0.47). Adding redundant views of the same content therefore causes a mild, graceful degradation, not an improvement. The model was trained with one sequence length (8×261 tokens). At larger V, each global-attention softmax spreads over more tokens, and the canonical scale (centroid of all views' points) becomes a harder global quantity to infer. Both are distribution shifts. The full-size VGGT gets better with more views because it is trained with a varying number of frames (2–24).

Coverage matters more than count. With window = 4V, few views are very close together. Rotation is then easy (AUC@15 0.76 at V = 2) and depth is easiest (AbsRel 0.088), but the translation direction is almost undetermined (translation AUC@30 0.05), because the baseline is tiny. With many views spread over up to 128 frames, the viewpoint changes far exceed the 32-frame training range. Everything then collapses: AbsRel 0.39 and rotation AUC@15 0.19 at V = 32. The accuracy curve is governed by how far the test-time baseline distribution is from training, more than by V itself.

Training with more views is better, even at fewer test views. Evaluated with 4 views, where the V = 4 model is in-distribution, the V = 8 model is still at least as good (AbsRel 0.124 vs 0.131, pose AUC@30 0.162 vs 0.130). It is clearly better at 8 views (0.139 vs 0.148, pose AUC@30 0.179 vs 0.126) and extrapolates better to 16–32 views. Each 8-view sample gives twice the depth supervision and 56 instead of 12 pose pairs, and the global attention learns to aggregate over more frames. Limitation: the comparison holds the number of samples and steps fixed, not compute. The V = 4 run processes half as many images and cost about 32% of the GPU-hours (5.4 vs 16.9). I also trained only one seed per setting.

Runtime and memory. With B = 1, the forward pass takes ~20 ms and stays flat up to 16 views (a 4,176-token sequence). The GPU is latency- and launch-bound for this small model, and the frozen ViT-B processes all frames as one batch. Beyond that, global attention (quadratic in V·N, but memory-efficient with SDPA) and the per-frame encoder start to dominate: 29 ms at 24 views and 38 ms at 32. Peak memory grows almost linearly, from 0.74 GB (mostly weights) to 1.05 GB at 32 views, because SDPA never materializes the (V·N)2 attention matrix. At inference, compute is therefore not the bottleneck for V ≤ 32. Accuracy is, because it depends on training with matching view counts and baselines. A natural next step is to train with a random V and window per batch.