1 Multi-view data preprocessing
Data check (scripts/check_data.py)
All checks pass on 4 training samples (224 samples in the split). Point and ray reprojection errors are about 10−4 px, well under the 0.01 px threshold. After canonicalization the valid-point centroid sits at distance 1.000 from the origin.
| Sequence | Views | Res. | Valid depth | Max point reproj. (px) | Max ray reproj. (px) | Centroid dist. |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 1.13e-04 | 1.09e-04 | 1.000 |
| 5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.23e-04 | 1.22e-04 | 1.000 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 9.78e-05 | 8.90e-05 | 1.000 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.17e-04 | 1.12e-04 | 1.000 |
Batch shapes
| Tensor | Shape |
|---|---|
| rgb | (2, 8, 3, 224, 224) |
| depth | (2, 8, 1, 224, 224) |
| depth_valid | (2, 8, 1, 224, 224) |
| K | (2, 8, 3, 3) |
| frame_numbers | (2, 8) |
| T_world_from_camera | (2, 8, 4, 4) |
| original_world_from_reference | (2, 4, 4) |
| scene_scale | (2) |
| points_world | (2, 8, 3, 224, 224) |
| ray_origins | (2, 8, 3, 224, 224) |
| ray_directions | (2, 8, 3, 224, 224) |
| plucker | (2, 8, 6, 224, 224) |
Unprojection
Canonicalization
The first camera sits at the origin looking down +z, and the scene is in normalized units.




Explanations
- Canonicalization
- Each ScanNet++ scene has its own arbitrary world origin and scale, so absolute poses don't mean much to a network. We move everything into the first camera's frame with
T′ᵢ = T₀⁻¹ Tᵢ, which puts camera 0 at the origin with identity rotation. Then we divide all translations, depths and points bys = ‖mean(valid points)‖, so the valid-point centroid ends up at distance 1. Every sample now shares the same origin, orientation and scale. - World-space ray directions
- For pixel
(u, v), the direction in camera coordinates isK⁻¹[u, v, 1]ᵀ, using the intrinsics after cropping and resizing. Rotating it into the world with the camera-to-world rotationRand normalizing givesd, and the ray starts at the camera centero = t. Since the world frame is the canonical first-camera frame, rays from every view share one coordinate system, which is what lets the network relate pixels across views. - Plücker moment
- Each ray becomes a 6-vector
(d, m), where the moment ism = o × d. Any point on the ray gives the same moment, since(o + λd) × d = o × d, so(d, m)describes the line itself, not a particular point on it. The size ofmis the line's distance from the origin. Stacked per pixel, this gives the network a 6-channel image that encodes the camera's intrinsics and pose.
2 Mini-LVSM novel-view synthesis
2.1 Overfitting one batch
After 2000 steps on one fixed batch, the final overfit PSNR is 30.96 dB, above the ~30 dB target.

2.2 Training and evaluation
Settings. Default configs/lvsm.json, unchanged: 14.6 M parameters, patch size 8, 224×224 inputs, 8 views per sample (2 source, 6 target), batch size 4, AdamW at lr 2·10⁻⁴, 100 epochs. The best validation PSNR (16.15 dB, SSIM 0.560) came at the last epoch, so the model was still improving.

Ablation over source views and ray conditioning (validation)
| Source views | Target rays | PSNR (dB) ↑ | SSIM ↑ | Images |
|---|---|---|---|---|
| S=1 | correct | 15.19 | 0.5583 | 196 |
| S=1 | shuffled | 11.80 | 0.4877 | 196 |
| S=1 | none | 12.47 | 0.5292 | 196 |
| S=2 | correct | 16.15 | 0.5603 | 168 |
| S=2 | shuffled | 11.49 | 0.4723 | 168 |
| S=2 | none | 12.44 | 0.5266 | 168 |
| S=4 | correct | 16.89 | 0.5754 | 112 |
| S=4 | shuffled | 11.59 | 0.4730 | 112 |
| S=4 | none | 12.45 | 0.5364 | 112 |
Each visualization row is one validation sample. Columns: the S source views, ground truth, then the prediction.









Discussion
- More source views help. With correct rays, PSNR rises from 15.19 (S=1) to 16.15 (S=2) to 16.89 dB (S=4), even though the model only ever saw S=2 during training. More views cover more of each target's camera, so less content has to be guessed, and attention generalizes across a different number of source tokens. With one view, much of the target is unobserved and the model can only hedge toward a blurry mean.
- Target rays carry the geometry. With shuffled rays (another target's rays), PSNR drops by about 3.4–5.3 dB and the model renders plausible images from the wrong viewpoint. With no rays, every prediction collapses to a flat, roughly average-coloured image, because the model doesn't know where to look. "None" scores slightly higher than "shuffled" because MSE penalizes a sharp image from the wrong viewpoint more than a flat average. Neither ablation improves with more source views, so the gains from S come from correct ray conditioning.
- Limitations. At ~16–17 dB the predictions capture coarse layout and colour (walls, floors, large furniture) but are blurry and show 8×8 patch artefacts. The model is small (14.6 M parameters) and trained for 100 epochs on ~200 scenes, and plain MSE favours blurry averages. The validation curve was still rising at epoch 99, so longer training would help.
2.3 Bonus: training with a random number of source views
Question. The baseline always trains with S=2 source views and is evaluated with 1, 2 and 4. Does drawing S uniformly from {1, 2, 3, 4} for each training batch (T = 8 − S targets) make the model more robust to the number of source views at test time?
Setting. Same model, data, optimizer, schedule, seed and 100-epoch budget as Q2.2 (configs/lvsm_randS.json). The only change is the per-batch random S. The checkpoint is selected by validation PSNR at S=2, as for the baseline, and both use best.pt with correct rays.
Results. Random-S training matches the baseline at every evaluation S. PSNR is +0.00 dB at S=1, +0.02 dB at S=2 and +0.10 dB at S=4, and every SSIM change is under 0.002. The validation curves at S=2 overlap for the whole run. Both models gain about 1.7 dB going from 1 to 4 source views, and the predictions look the same (blurry, patchy colour layouts).
Conclusions and explanation. Training with a random number of source views did not really help. (1) The baseline already handles different S well. The transformer just sees a set of source tokens, each tagged with its own rays, so it doesn't care much how many there are, and both models get better with more views. (2) The only small gain is at S=4 (+0.10 dB), which makes sense because the random-S model saw 3 and 4 views during training. At S=1 nothing changes, because the problem there is that one view simply doesn't show enough of the scene. (3) The random-S model also sees slightly fewer target views per step. At this size, the model's capacity and the short 100-epoch training matter more than which S it trained with.
Limitations. One seed per configuration, so differences of about 0.1 dB are within run-to-run noise. Checkpoints were selected at S=2, which favours the baseline's training setting. Only 100 epochs were run, and the gap could change for a longer-trained model that has learned to rely more on cross-view correspondence.
| Eval S | PSNR base | PSNR rand-S | Δ | SSIM base | SSIM rand-S | Δ |
|---|---|---|---|---|---|---|
| S=1 | 15.19 | 15.19 | +0.00 | 0.5583 | 0.5588 | +0.0004 |
| S=2 | 16.15 | 16.16 | +0.02 | 0.5603 | 0.5597 | -0.0006 |
| S=4 | 16.89 | 16.99 | +0.10 | 0.5754 | 0.5738 | -0.0016 |

3 Mini-VGGT feed-forward geometry
3.1 Overfitting one scene

3.2 Training and evaluation
Settings. Default configs/vggt.json, unchanged: frozen DINOv2 ViT-B/14 encoder with 57.7 M trainable parameters, 8 views per sample from a 32-frame window, 224×224, batch size 16, AdamW at lr 6·10⁻⁴, 1000 epochs, loss = 5·camera + depth.

Validation and test metrics
Metrics from best.pt on the 28 validation scenes and the 28 held-out test scenes (separate splits, neither used for training).
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.1437 | 0.1547 |
| Depth δ1 | 0.8321 | 0.8021 |
| Rotation error (°) | 12.56 | 9.29 |
| Translation-dir. error (°) | 42.34 | 41.06 |
| Rotation AUC@5 | 0.1931 | 0.2041 |
| Rotation AUC@15 | 0.5236 | 0.5386 |
| Rotation AUC@30 | 0.7045 | 0.7306 |
| Translation AUC@5 | 0.0099 | 0.0084 |
| Translation AUC@15 | 0.0754 | 0.0504 |
| Translation AUC@30 | 0.1885 | 0.1593 |
| Pose AUC@5 | 0.0043 | 0.0023 |
| Pose AUC@15 | 0.0559 | 0.0344 |
| Pose AUC@30 | 0.1622 | 0.1451 |
GT vs. prediction on four held-out validation scenes
Each pair shows the same viewpoint. Left: GT cameras + GT depth unprojected. Right: predicted cameras + predicted depth, in the same canonical frame and colours.
Discussion of held-out errors
- Depth generalizes well. Validation AbsRel is 0.144 and δ1 is 0.832 (test 0.155 / 0.802). The frozen DINOv2 features already encode strong monocular cues. The reconstructions get room layout, walls, floor and large furniture right, but depth edges are smoothed, thin structures (chair legs, table edges) bleed into the background, and planes are slightly bent. The depth head predicts each 14×14 patch from a single token and only a small RGB-guided convolution refines it at full resolution, so fine detail inside a patch is limited.
- Rotation is fair, translation direction is the weak point. The mean relative rotation error is 12.6° (AUC@30 0.70), but the relative translation-direction error is about 42° (AUC@30 0.19). Pose AUC takes the worse of the two, so it is limited by translation (0.16 at 30°). Within a 32-frame window, baselines are short compared with scene depth, so the direction of a small translation is poorly conditioned. Therefore, a small absolute position error becomes a large angular one.
- Data-limited generalization. With only 224 training samples per epoch (~200 scenes), the train loss stays above the validation loss, and the validation camera loss flattens in the last ~150 epochs. Test numbers are close to validation, so the model is not overfitting to the validation scenes. It is capacity- and data-limited.
3.3 Bonus: how many input views?
Question. The model was trained with exactly V=8 views. At test time, how do accuracy, runtime and memory change when it receives 2 to 32 views of the same scene window?
Setting. One trained checkpoint (outputs/vggt_train/best.pt), validation split, the same 32-frame prepared windows (seed 825), and V ∈ {2, 4, 8, 16, 24, 32} views sampled from each window. Batch size 1. Runtime is the model forward pass (encoder + aggregator + heads, bf16), averaged over 27 samples after one warm-up. Memory is the peak allocated GPU memory. Pairwise AUCs are computed over all V(V−1) pairs, so a larger V also means more (and more widely spaced) pairs. Script: tools_report/vggt_views.py.
| V | AbsRel ↓ | δ1 ↑ | Rot AUC@30 ↑ | Trans AUC@30 ↑ | Pose AUC@30 ↑ | ms / sample | Peak mem (GB) |
|---|---|---|---|---|---|---|---|
| 2 | 0.1132 | 0.8702 | 0.750 | 0.201 | 0.181 | 29.6 | 0.98 |
| 4 | 0.1315 | 0.8508 | 0.697 | 0.192 | 0.167 | 30.7 | 0.99 |
| 8 | 0.1437 | 0.8321 | 0.704 | 0.189 | 0.162 | 38.6 | 1.03 |
| 16 | 0.1679 | 0.7983 | 0.668 | 0.168 | 0.142 | 66.0 | 1.08 |
| 24 | 0.1697 | 0.7763 | 0.668 | 0.168 | 0.143 | 104.7 | 1.37 |
| 32 | 0.1737 | 0.7694 | 0.659 | 0.164 | 0.139 | 148.8 | 1.43 |

Results. More input views make every accuracy metric slightly worse. Depth AbsRel goes from 0.113 (V=2) to 0.144 (V=8) to 0.174 (V=32), and δ1 from 0.870 to 0.769. Rotation AUC@30 drops from 0.75 to 0.66, while translation AUC@30 stays flat at about 0.16–0.20, so pose AUC@30 goes from 0.18 to 0.14. Runtime is nearly constant up to V=8 (30–39 ms) and then grows faster than linearly (66 ms at 16, 149 ms at 32). Peak memory only rises from 0.98 to 1.43 GB, because at batch size 1 under no_grad most of it is the 144 M-parameter weights.
Explanations. (1) Train/test mismatch: the model only saw V=8. With V=32 the global-attention layers attend over 4× more tokens (32 × 256), which changes the attention statistics the camera and depth tokens were trained on. The decline is steady up to V=16 and then levels off. (2) Harder views: with V=2, one of the two views is the reference frame, which has the identity pose and is the easiest to predict depth for. Larger V adds more frames far from the reference within the 32-frame window, whose poses and scale must be inferred through the chain of views. So the metrics are not computed on identical image sets, and part of the trend comes from the views, not the model. (3) Cost: frame attention and the DINO encoder scale linearly in V, global attention quadratically in V·N. At small V, fixed overheads (kernel launches, heads) dominate, so the curve is flat. Beyond 8 views the quadratic global attention starts to show. Memory stays low thanks to the small model and inference without gradients. Training at V=32 with batch 16 would be a different story: activations would grow about 4× and would not fit in 16 GB.
Conclusions and limitations. Giving this small model more views at test time doesn't make it more accurate. It was only trained with 8 views, so it hasn't learned to use the extra views the way the full VGGT does, which trains with varying view counts.

















