16-825: Learning for 3D Vision · Kashu Yamazaki
s = sizeout/sizein per axis,
f' = s·f and, from xout+0.5 = s(xin+0.5),
c' = s(c+0.5) − 0.5 (matching the align_corners=False resize).(u, v, 1) are mapped by
K−1 to rays with z = 1; multiplying by the z-depth gives camera
points, and Xw = R Xc + t gives world points.T'i = T0−1 Ti (float64), so camera 0 is
the identity. Points are unprojected in this frame, the scale s = ‖mean(pvalid)‖ comes from
the provided compute_scene_scale, and translations, depth and points are divided by s.d = normalize(R K−1(u, v, 1)T), origin = camera centre t.[d, o × d].Points from all 8 views overlap into one consistent room, so the depth unprojection and the camera-to-world poses agree.



The text panel on the right of each screenshot shows the first camera centre (0, 0, 0) and a valid-point centroid distance of 1.000.



| Sample | Valid depth | Max reproj. err (px) | Max ray reproj. err (px) | Centroid dist. |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.978 | 1.1e-04 | 1.1e-04 | 1.000 |
| 5f99900f09_frame_002820 | 0.990 | 1.2e-04 | 1.1e-04 | 1.000 |
| 47b37eb6f9_frame_002720 | 0.961 | 9.8e-05 | 8.9e-05 | 1.000 |
| 4c5c60fa76_frame_006690 | 0.752 | 1.1e-04 | 1.0e-04 | 1.000 |
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011141905270051211,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.777836385183036e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011162471491843462,
"max_ray_reprojection_error_px": 0.00010225349979009479,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}Canonicalization. Absolute ScanNet++ poses live in an arbitrary per-scene world frame and
metric scale, which the model cannot observe from images. Re-expressing every pose relative to the
first sampled camera (T0−1Ti) removes the arbitrary
rotation/translation, and dividing by the distance to the valid-point centroid removes the unobservable
global scale. Every sample then has camera 0 at the identity looking down +Z, with scene content about one unit away,
so targets have a consistent, learnable distribution. Only one scale factor is used, so relative
geometry is preserved, and the centroid is not subtracted, so camera 0 stays at the origin.
World-space ray directions. A pixel's camera-space direction is K−1(u, v, 1).
Rotating it by the camera-to-world rotation R (no translation, because directions are
vectors) and normalizing gives the unit direction in the canonical frame. All views therefore share
one coordinate system, and the rays encode both the intrinsics (ray spread) and the orientation of each camera.
Plücker moment. A ray through origin o with direction d has moment
m = o × d. The moment is the same for any point o + λd on the line
(since d × d = 0), so [d, m] describes the line itself, not a particular origin.
|m| is the distance from the world origin to the line, and the direction of m is the normal of the plane
containing the line and the origin. Camera 0's rays all have m = 0, because its centre is the origin.
Per-pixel Plücker maps give the transformer a dense, smooth camera encoding.
(..., C, H, W) → (..., (H/p)(W/p), p·p·C) by
reshaping into a patch grid and permuting so each token holds one p×p×C block
(row-major over the grid). unpatchify is its exact inverse (checked: round-trip is bit-exact).[2·rgb−1, Plücker] (9 channels);
target tokens = linear projection of patchified target Plücker rays (6 channels). The same fixed 2D sin-cos
position embedding is added to every view's tokens. All S·N source tokens are repeated for each of the
T targets and concatenated with that target's N tokens, giving a
(B·T, (S+1)N, d) sequence. Targets are therefore rendered independently, and in parallel.N (target) tokens →
LayerNorm → linear RGB head → sigmoid → unpatchify to (B, T, 3, H, W).[0, 1], computed in float32.
The training step runs the forward pass under bf16 autocast, then backward, gradient clipping, the optimizer step and the scheduler step.Final overfit validation PSNR = 30.94 dB (SSIM 0.871, MSE 8.6e-4) after 2000 steps on one fixed batch of 4 sequences (S = 2, T = 6). With the default batch size of 4, the memorization check passes the ≥ 30 dB target. (A first attempt with the larger batch of 8 that I use for full training reached 28.5 dB in the same 2000 steps, because it has twice as many images to memorize.)
val_epoch000.png from the overfit run. Columns: 2 source views, ground truth, prediction.| Setting | Value |
|---|---|
| Model | patch 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm (14.6M parameters) |
| Data | 224 train scenes, 8 views sampled from a 32-frame window, S = 2 sources / T = 6 targets, 224×224 |
| Optimisation | AdamW, lr 2e-4, betas (0.9, 0.95), weight decay 0.05, 500 warmup steps, then cosine decay to 10%; grad clip 1.0 |
| Schedule | batch 8 sequences (48 target images), 400 epochs = 11.2k steps, bf16 autocast |
| Hardware | 1× NVIDIA L40S, ~18 s/epoch, about 2 h in total |
| Selection | best.pt by validation PSNR (prepared val split, first 32 frames) |
| Sources | Correct PSNR | SSIM | Shuffled PSNR | SSIM | None PSNR | SSIM |
|---|---|---|---|---|---|---|
| S = 1 | 15.86 | 0.586 | 11.71 | 0.483 | 12.40 | 0.518 |
| S = 2 | 17.09 | 0.595 | 11.28 | 0.461 | 12.42 | 0.515 |
| S = 4 | 18.20 | 0.614 | 11.33 | 0.457 | 12.45 | 0.522 |
Test split with true rays: S = 1: 16.25 dB / 0.566, S = 2: 17.21 dB / 0.578, S = 4: 18.47 dB / 0.585.

val_epoch000.png (epoch 0)
val_epoch380.png (epoch 380)









Number of source views. The model was trained only with S = 2, yet quality still increases monotonically with more source views at test time: 15.86 → 17.09 → 18.20 dB PSNR for S = 1 → 2 → 4. The source tokens are simply concatenated, so the transformer can attend to more context for free. With one view, much of the target is unobserved, and the model produces a blurry mean colour there. With four views, most target content has been seen from some source, so the layout and colours are copied more faithfully.
Ray conditioning. The target Plücker rays are the only signal saying where to render. With
shuffled rays, PSNR falls by about 5.8 dB (S = 2), even below none. The model confidently renders a
plausible image from the wrong viewpoint, and the misaligned image is penalized more heavily than a hedge. With
none (zero rays), the model receives an out-of-distribution query and outputs a blurry average of the
sources (about 12.4 dB regardless of S). In both ablations, extra source views no longer help, which confirms that the model really
uses the camera geometry rather than copying the nearest source. The correct-ray predictions are spatially
right but blurry, with visible 8×8 patch seams, as expected for a 14.6M-parameter MSE-trained model
after 2 GPU-hours. MSE favours the conditional mean under uncertainty.
Question. The baseline always trains with S = 2 source views, but it is evaluated with 1, 2 and 4. Does training with a random number of source views per step, S ~ U{1, 2, 3, 4}, make the model better at every evaluation S, and in particular at S = 1 and S = 4, which the baseline never sees?
Setup. I added an optional train_num_source_views list to the config. At each training step,
train_one_epoch draws S from it, and the remaining T = 8 − S views are the targets
(configs/lvsm_randS.json). Everything else is identical to the baseline: model, data, batch 8, 400 epochs / 11.2k steps,
learning-rate schedule and data seed. Per-epoch validation and best.pt selection still use S = 2. Both models are evaluated with
the same models.lvsm.eval ablation grid. Since T = 8 − S, the random-S model sees slightly fewer target images per step
(5.5 vs 6 on average) but more context per target.
| Eval setting | Fixed S=2 PSNR | Random S PSNR | Δ | Fixed SSIM | Random SSIM |
|---|---|---|---|---|---|
| S = 1, correct | 15.86 | 15.92 | +0.06 | 0.586 | 0.590 |
| S = 2, correct | 17.09 | 17.21 | +0.12 | 0.595 | 0.600 |
| S = 2, shuffled | 11.28 | 11.23 | -0.05 | 0.461 | 0.460 |
| S = 2, none | 12.42 | 12.40 | -0.02 | 0.515 | 0.524 |
| S = 4, correct | 18.20 | 18.42 | +0.21 | 0.614 | 0.622 |




Results. Random-S training improves every correct-ray setting, and the gain grows with S: +0.06 dB at S = 1, +0.12 dB at S = 2 and +0.21 dB at S = 4 (SSIM +0.004 / +0.005 / +0.007). The S = 2 validation curves overlap for most of training, and the random-S model only pulls slightly ahead in the last ~100 epochs. The ablations with shuffled or no rays stay at the same low level (±0.1 dB), so the extra quality comes from better use of correctly-posed sources, not from a different prior.
Interpretation. The biggest gain is at S = 4, where the baseline is out of distribution: it has never had to attend over four source views, and random-S training makes long-context aggregation part of training. At S = 1, the gain is small even though the model trained with single views 25% of the time. There, the error is dominated by content that is simply not visible in the one source view, and a deterministic MSE model can only predict a blurry mean for it, however it was trained. Training with more sources on average (2.5) may also give cleaner gradients for learning ray-to-pixel correspondence, which also helps S = 2.
Limitations. The differences are small (0.1–0.2 dB) and come from a single seed per setting, so only the trend (gain increasing with S) should be trusted, not the exact numbers. The runs used the same number of steps, but the random-S run ran on a slower GPU (A6000, 4.1 h vs 2.1 h on an L40S). Both models are still improving at 400 epochs, so the gap at convergence may differ. A curriculum, or sampling larger S more often, could be explored next.
aggregate). Frozen DINOv2 ViT-B/14 patch tokens are projected to width 384, and each
frame gets 1 camera + 4 register tokens prepended (a separate learned set for the reference frame 0, which defines the canonical frame). Even-indexed blocks run frame attention on (B·V, N, d), so
tokens only see their own image. Odd-indexed blocks run global attention on (B, V·N, d), across all views. Both
use 2D RoPE. Positions are repeated V times for the global sequence, so a patch keeps its
in-image position. The output of the last frame block is concatenated with the final global output along channels, then
normalized, giving the heads 2d-dimensional per-frame and cross-view features.T and K.[coarse, RGB] adds a residual, then depth = exp(·). A separate projection gives the confidence
c = 1 + exp(·) (always > 1).c·|d−d̂| − α log c (α = 0.2), plus a plain trimmed
L1 regression term (both drop the worst 2% of pixels) and a four-scale gradient-matching term on x/y finite differences of the residual (masked by validity, clamped).
Total = 5·camera + 1·depth.After 40 epochs (1000 steps) on one fixed scene, the model fits it almost exactly: depth AbsRel 0.036, δ1 0.956, mean rotation error 0.31°, mean translation-direction error 0.29°, rotation AUC@5 1.000, translation AUC@5 0.986, joint pose AUC@5 0.986 (AUC@30 0.998). The total loss goes negative because the −α log c term dominates once the depth error is small.



| Setting | Value |
|---|---|
| Model | Frozen DINOv2 ViT-B/14 encoder (224 px, 16×16 patches); aggregator width 384, 6 heads, 8 alternating blocks (4 frame + 4 global) |
| Data | 8 views sampled from a 32-frame window, 224×224, 224 training scenes (unchanged defaults) |
| Optimisation | AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs, then cosine decay to 5%; loss weights camera 5 / depth 1 |
| Schedule | Global batch 16 (4 per GPU), 1000 epochs = 14k steps, bf16 autocast |
| Hardware | 4× NVIDIA L40 with DDP (torchrun), 4 h 13 min in total (~15 s/epoch including validation) |
| Selection | best.pt = lowest validation loss (epoch 896 of 1000) |

| Split | AbsRel ↓ | δ1 ↑ | Rot@5 | Rot@15 | Rot@30 |
|---|---|---|---|---|---|
| Validation | 0.1388 | 0.836 | 0.198 | 0.519 | 0.704 |
| Test | 0.1525 | 0.795 | 0.192 | 0.536 | 0.735 |
| Split | Trans@5 | Trans@15 | Trans@30 | Pose@5 | Pose@15 | Pose@30 |
|---|---|---|---|---|---|---|
| Validation | 0.018 | 0.088 | 0.201 | 0.005 | 0.069 | 0.179 |
| Test | 0.006 | 0.065 | 0.185 | 0.002 | 0.048 | 0.168 |
| Split | Loss | Cam loss | Depth loss | Mean rot. err (°) | Mean trans. err (°) | Depth RMSE (norm.) | Pose pairs |
|---|---|---|---|---|---|---|---|
| Validation | 0.809 | 0.110 | 0.259 | 12.47 | 40.44 | 0.191 | 784 |
| Test | 0.752 | 0.099 | 0.259 | 9.37 | 40.06 | 0.197 | 784 |
Left: GT cameras and depth; right: predicted cameras and depth, from the provided viewer
(visualization/index.html).








Depth generalizes reasonably well: AbsRel is 0.139 (val) / 0.153 (test), and 84% / 80% of pixels fall within 1.25× of GT. The screenshots show that the coarse room layout and the global scale are recovered: walls, floor and tables sit in the right place. The predicted depth maps are much smoother than GT, though. Thin structures (chair legs in 09c1414f1b, the chair in 21d970d8de) are averaged into their background. Depth discontinuities are blurred, so the point clouds show "flying" points between foreground and background. Long, low-texture hallways (280b83fcf3) are compressed: the far end is predicted too close. These are the typical failure modes of a dense head on frozen 14-pixel patches, which uses only a light RGB refinement, under an L1/confidence loss that favours the median depth.
Rotations are mostly right: rotation AUC@30 is 0.70 (val) / 0.74 (test), with a mean error of 12.5° / 9.4°. Trajectories like 2b1dc6d6a5 are recovered almost exactly. Fine accuracy is still weak (AUC@5 ≈ 0.2). Translation direction is the main error source: the mean direction error is about 40°, and translation AUC@30 is only 0.20 / 0.18. The joint pose AUC, which takes the worse of the two errors, is therefore limited by translation. Within a 32-frame window, many handheld iPhone baselines are short compared with the scene depth. The direction of such a small translation is ill-conditioned: a tiny absolute position error becomes a large angular error. In 09c1414f1b, one far camera (frame 290) is pulled into the cluster of nearby cameras. The model was also still improving when training ended: the validation loss kept decreasing until epoch 896, and the camera loss had not plateaued. A longer schedule, unfreezing DINO, or more than 224 training scenes should help most.
Val vs. test. The test split has slightly worse depth (AbsRel +0.014) but better rotation. Both splits contain only 28 scenes, so these differences are within scene-to-scene variance. There is no sign of overfitting: the validation loss was still decreasing at the end, so the model is capacity- and schedule-limited.
Question. Mini-VGGT uses global attention over the tokens of all views. How do accuracy, runtime and memory depend on the number of input views V, both at evaluation time and during training?
Setup. Two models with the same architecture and hyperparameters (global batch 16, 1000 epochs, same schedule) differ
only in the number of training views: the main model (trained V = 8) and a second model (trained V = 4, 2× L40, 2 h 41 min).
Each is evaluated on the full validation split with V ∈ {2, 4, 8, 16, 24, 32}, in two regimes
(scripts/vggt_views_study.py). The prepared loader always keeps frame 0 as the reference and samples the other V−1 frames from the window with seed 825.
Runtime and peak memory are measured for one sample (B = 1, bf16, including the frozen encoder and the weights) on an L40S, after warm-up. Depth metrics average over all pixels of all V frames, and pose AUCs over all V(V−1) ordered pairs, so V = 2 has only 56 pairs and its pose numbers are noisy.
| Trained | Eval views | AbsRel | δ1 | Rot@15 | Pose@15 | Pose@30 | Forward (ms) | Peak mem (GB) |
|---|---|---|---|---|---|---|---|---|
| V=8 | 2 | 0.1102 | 0.871 | 0.505 | 0.064 | 0.205 | 19.8 | 0.74 |
| V=8 | 4 | 0.1240 | 0.855 | 0.500 | 0.056 | 0.162 | 20.1 | 0.76 |
| V=8 | 8 | 0.1388 | 0.836 | 0.519 | 0.069 | 0.179 | 20.5 | 0.80 |
| V=8 | 16 | 0.1659 | 0.806 | 0.473 | 0.046 | 0.144 | 20.4 | 0.89 |
| V=8 | 24 | 0.1694 | 0.790 | 0.473 | 0.047 | 0.148 | 29.4 | 0.97 |
| V=8 | 32 | 0.1755 | 0.782 | 0.472 | 0.048 | 0.149 | 38.2 | 1.05 |
| V=4 | 2 | 0.1111 | 0.884 | 0.495 | 0.036 | 0.105 | – | – |
| V=4 | 4 | 0.1313 | 0.856 | 0.527 | 0.045 | 0.130 | – | – |
| V=4 | 8 | 0.1481 | 0.833 | 0.497 | 0.036 | 0.126 | – | – |
| V=4 | 16 | 0.1825 | 0.779 | 0.435 | 0.025 | 0.102 | – | – |
| V=4 | 24 | 0.1845 | 0.754 | 0.437 | 0.026 | 0.104 | – | – |
| V=4 | 32 | 0.1906 | 0.749 | 0.420 | 0.026 | 0.099 | – | – |
Timing and memory are listed only for the V = 8 model (L40S). The V = 4 study ran on a slower L40, and the architecture is identical.
| Eval views | Frame window | AbsRel | δ1 | Rot@15 | Pose@15 | Pose@30 |
|---|---|---|---|---|---|---|
| 2 | 8 | 0.0876 | 0.927 | 0.760 | 0.021 | 0.052 |
| 4 | 16 | 0.1104 | 0.891 | 0.622 | 0.057 | 0.136 |
| 8 | 32 | 0.1388 | 0.836 | 0.519 | 0.069 | 0.179 |
| 16 | 64 | 0.2252 | 0.662 | 0.330 | 0.047 | 0.130 |
| 24 | 96 | 0.3452 | 0.548 | 0.221 | 0.024 | 0.090 |
| 32 | 128 | 0.3894 | 0.438 | 0.185 | 0.017 | 0.071 |
More views at test time do not help this model. In the fixed window, the V = 8 model's depth AbsRel rises from 0.110 (V = 2) to 0.139 (V = 8) and 0.176 (V = 32), while rotation AUC@15 stays roughly flat (0.51 → 0.47). Adding redundant views of the same content therefore causes a mild, graceful degradation, not an improvement. The model was trained with one sequence length (8×261 tokens). At larger V, each global-attention softmax spreads over more tokens, and the canonical scale (centroid of all views' points) becomes a harder global quantity to infer. Both are distribution shifts. The full-size VGGT gets better with more views because it is trained with a varying number of frames (2–24).
Coverage matters more than count. With window = 4V, few views are very close together. Rotation is then easy (AUC@15 0.76 at V = 2) and depth is easiest (AbsRel 0.088), but the translation direction is almost undetermined (translation AUC@30 0.05), because the baseline is tiny. With many views spread over up to 128 frames, the viewpoint changes far exceed the 32-frame training range. Everything then collapses: AbsRel 0.39 and rotation AUC@15 0.19 at V = 32. The accuracy curve is governed by how far the test-time baseline distribution is from training, more than by V itself.
Training with more views is better, even at fewer test views. Evaluated with 4 views, where the V = 4 model is in-distribution, the V = 8 model is still at least as good (AbsRel 0.124 vs 0.131, pose AUC@30 0.162 vs 0.130). It is clearly better at 8 views (0.139 vs 0.148, pose AUC@30 0.179 vs 0.126) and extrapolates better to 16–32 views. Each 8-view sample gives twice the depth supervision and 56 instead of 12 pose pairs, and the global attention learns to aggregate over more frames. Limitation: the comparison holds the number of samples and steps fixed, not compute. The V = 4 run processes half as many images and cost about 32% of the GPU-hours (5.4 vs 16.9). I also trained only one seed per setting.
Runtime and memory. With B = 1, the forward pass takes ~20 ms and stays flat up to 16 views (a 4,176-token sequence). The GPU is latency- and launch-bound for this small model, and the frozen ViT-B processes all frames as one batch. Beyond that, global attention (quadratic in V·N, but memory-efficient with SDPA) and the per-frame encoder start to dominate: 29 ms at 24 views and 38 ms at 32. Peak memory grows almost linearly, from 0.74 GB (mostly weights) to 1.05 GB at 32 views, because SDPA never materializes the (V·N)2 attention matrix. At inference, compute is therefore not the bottleneck for V ≤ 32. Accuracy is, because it depends on training with matching view counts and baselines. A natural next step is to train with a random V and window per batch.