Q1

1.2 Unprojection visualization

Unprojection: original world coordinates

1.3 Canonicalization visualization

Canonicalization: first camera at the origin, centroid distance 1.000

1.7 Final Check

Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Canonicalization: Every pose is multiplied by the inverse of camera 0's pose, so camera 0 becomes the identity at the origin. Then translations, depths, and points are divided by the distance from camera 0 to the centroid of the valid points, which puts the centroid at distance 1. This removes the arbitrary world origin, orientation, and metric scale of each scan, so all scenes share one normalized frame. The scale is stored separately so evaluation can undo it.

World-space ray directions: For pixel (u, v) the camera-frame direction is K^-1 [u, v, 1]. Rotating it by the camera-to-world rotation and normalizing gives a unit direction in the shared frame. The ray origin is the camera center. These define the 3D line each pixel looks along, which lets the model relate pixels across views.

Plücker moment: A line has no natural starting point, so Plücker coordinates [d, m] describe it without one: d is the unit direction and m = o × d is the moment. Moving the origin along the line changes o by a multiple of d, which leaves m unchanged. The moment's length is the line's distance from the world origin and its direction is the normal of the plane through the line and the origin. The 6 values per pixel are the camera conditioning for the Mini-LVSM tokens.

Q2

2.1 overfit PSNR: 30.929217020670574 val_epoch000

2.2 Training and Evaluation

Training settings: default configs/lvsm.json. Patch size 8, width 384, 8 transformer layers, head dim 64, MLP ratio 4, QK-norm. 8 views per sample at 224 px (sampling range 32), 2 source views and 6 targets, batch size 4, 100 epochs (5600 steps), AdamW with lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps, cosine decay, grad clip 1.0, bf16 autocast with the MSE loss in float32. Training took 17.5 minutes on one RTX 5090.

Training curves (train and validation, per epoch):

Mini-LVSM training curves

Final epoch: train PSNR 15.60 dB, validation PSNR 16.13 dB, SSIM 0.560. Validation is slightly above train because training batches use random view sampling and the train number is averaged while the weights are still changing within the epoch; there is no overfitting, and both curves are still rising at epoch 100, so the model is underfit.

Validation ablations (python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none):

Source views Conditioning PSNR (dB) SSIM
1 correct 15.19 0.558
1 shuffled 11.80 0.487
1 none 12.45 0.526
2 correct 16.13 0.560
2 shuffled 11.49 0.472
2 none 12.43 0.525
4 correct 16.87 0.575
4 shuffled 11.58 0.472
4 none 12.44 0.536

Visualizations. Columns: source views, ground truth, prediction. One row per target view.

Training visualization (validation batch at epoch 95, S = 2):

Training visualization, epoch 95

Evaluation, number of source views (correct rays), S = 1 / 2 / 4:

Eval S=1 correct Eval S=2 correct Eval S=4 correct

Evaluation, ray conditioning at S = 2, shuffled / none:

Eval S=2 shuffled Eval S=2 none

Number of source views: quality improves monotonically with more sources (15.19, 16.13, 16.87 dB), even though the model was trained with 2. Extra views cover more of the target region and give the attention more evidence for each query ray, with diminishing returns from 2 to 4. The model generalizes to a sequence length it never saw in training because the token layout is the same, just longer.

Ray conditioning: the target rays are essential. Shuffled rays drop PSNR by about 4.6 dB and zero rays by about 3.7 dB, a larger effect than the source count. Shuffled is worse than none: with wrong rays the model confidently renders the wrong viewpoint, while with no rays it falls back to a blurry average of the sources, which scores higher under MSE. The gap shows that the model is not simply copying sources but reading the Plücker rays to decide which content lands in each patch.

Predictions are blocky and low-detail at every setting. With 8x8 patches regressed directly to pixels and only 5600 training steps, the model captures layout and color but not texture, consistent with the reference numbers (about 15 / 16 / 17 dB for S = 1 / 2 / 4).

2.3 Extra Credit: model capacity (patch size)

Question: how does the patch size affect quality and training cost? Doubling the patch from 8 to 16 cuts the token count per sequence by 4x (2352 to 588) and the attention cost by about 16x, but each token must then regress a 16x16x3 = 768-value pixel block instead of 192.

Settings: configs/lvsm_patch16.json is the 2.2 config with only patch_size changed from 8 to 16. Same data, width 384, 8 layers, 2 source views, batch 4, 100 epochs, lr 2e-4, bf16. Both runs on one RTX 5090 with 4 data-loader workers. Parameters: 14.6M (patch 8) vs 16.0M (patch 16); the difference is the larger input and output projections.

Results (validation, best.pt):

Patch 8 (baseline) Patch 16
PSNR / SSIM, S=1 correct 15.19 / 0.558 15.02 / 0.554
PSNR / SSIM, S=2 correct 16.13 / 0.560 15.89 / 0.555
PSNR / SSIM, S=4 correct 16.87 / 0.575 16.52 / 0.567
PSNR, S=2 shuffled / none 11.49 / 12.43 11.49 / 12.37
Tokens per sequence 2352 588
Median time per epoch 11.2 s 7.3 s
Total training time 17.5 min 11.3 min

Validation curves of the two runs:

Patch 8 vs patch 16 validation curves

Evaluation at S = 2 with correct rays, patch 8 (top) vs patch 16 (bottom). Columns: sources, ground truth, prediction.

Patch 8, S=2 Patch 16, S=2

Conclusions. Patch 16 loses only 0.2 to 0.35 dB PSNR and about 0.005 SSIM at every source count, while training 35% faster. The curves have the same shape throughout, so the gap is a capacity effect rather than a convergence-speed effect. The ablation trends are unchanged: more sources still help, and shuffled or missing rays hurt by the same 4 dB, so the model's use of ray conditioning does not depend on the patch size. Visually, patch 16 shows a coarser 14x14 block grid with visible seams, while patch 8 produces finer blocks; both capture layout and color but no texture. At this training budget the model is underfit, so the resolution lost to larger patches matters less than it would for a well-trained model, which explains the small gap.

The speedup is much smaller than the 16x drop in attention cost. GPU utilization during the patch 16 run was only 10 to 20%, so data loading (8 views per sample, geometry computed on the CPU) and the per-epoch validation pass set a floor on the epoch time.

Limitations: one seed per setting, so differences of a few tenths of a dB are near the noise level. Both runs use the same number of epochs rather than the same wall-clock budget; with equal time, patch 16 could train for 150 epochs and might close the gap. Peak GPU memory was not recorded. Only two patch sizes were tested; patch 4 was skipped because it would need 9408 tokens per sequence and hours of training.

Q3

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit: one fixed training clip of 8 views, 40 epochs of 25 steps, lr 1e-3.

Overfit loss curve: total, camera, depth

All three losses fall steadily. Over the 40 epochs the camera loss goes from 0.342 to 0.002 and the depth loss from 3.77 to -0.18 (negative because the confidence term subtracts 0.2 log(conf) once the error is small). On the clip itself depth AbsRel reaches 0.037 with δ1 = 0.954, and the mean rotation and translation errors fall from 50° / 90° to 0.2° / 0.5°, so the model, heads and losses can memorize geometry when asked to.

Ground truth vs predicted geometry on the overfit clip, same viewpoint. The prediction's depth map shows the 14x14 patch grid of the depth head; the cloud and the 8 frustums match GT closely (predicted camera 0 at (-0.0001, 0.0001, -0.0001), centroid distance 0.999).

Overfit clip: ground truthOverfit clip: predicted camera + depth
No description has been provided for this imageNo description has been provided for this image

3.2 Training and Evaluation

Training settings: default configs/vggt.json. Frozen DINOv2 ViT-B/14 encoder, width 384, 6 heads, 8 frame/global pairs (16 blocks), 4-block camera head, depth head with RGB refinement. 8 views per sample at 224 px, sampling range 32, batch 16 (14 steps per epoch), 1000 epochs, AdamW lr 6e-4 with 10 warmup epochs and cosine decay to 5%, weight decay 0.01, loss weights camera 5.0 and depth 1.0, bf16 autocast. 2.5 hours on one RTX 5090. best.pt is epoch 742 by validation loss (0.852).

Training loss curve: total, camera, depth

Both splits evaluated on 8 views from the first 32 prepared frames, 224 px, seed 825, 28 samples each:

Metric Validation Test
Depth AbsRel (lower is better) 0.152 0.161
Depth δ1 0.804 0.791
Rotation AUC @5° / 15° / 30° 0.185 / 0.525 / 0.717 0.191 / 0.508 / 0.705
Translation AUC @5° / 15° / 30° 0.019 / 0.090 / 0.188 0.006 / 0.065 / 0.178
Joint pose AUC @5° / 15° / 30° 0.004 / 0.065 / 0.165 0.003 / 0.045 / 0.157
Mean rotation / translation error 13.0° / 41.7° 10.2° / 41.9°

Ground truth (left) vs prediction (right) on four validation scenes, same viewpoint per scene:

Scene 1: ground truthScene 1: predicted camera + depth
No description has been provided for this imageNo description has been provided for this image
Scene 2: ground truthScene 2: predicted camera + depth
No description has been provided for this imageNo description has been provided for this image
Scene 3: ground truthScene 3: predicted camera + depth
No description has been provided for this imageNo description has been provided for this image
Scene 4: ground truthScene 4: predicted camera + depth
No description has been provided for this imageNo description has been provided for this image

Discussion of held-out errors. Validation and test numbers are close, so the model generalizes equally to both; the gap to the overfit clip (AbsRel 0.04 vs 0.15) is the usual generalization gap, and the loss curve shows the validation loss still improving slowly at 1000 epochs, so the model is capacity- or data-limited rather than overfit. Depth is the strong output: 80% of pixels are within 25% of GT, and the predicted clouds have the right room layout, with errors concentrated at depth discontinuities, thin structures and far walls, where predictions are smoothed. Rotation is reasonable (median error around 10°), but translation direction is the weak point with a mean error of about 42°. The clips are short (8 frames from 32), so the baselines between cameras are small relative to the scene scale and the translation direction is poorly constrained; a few degrees of rotation error moves the camera center by a comparable amount. Because the joint AUC takes the worse of the two errors per pair, it is dominated by translation. In the viewers this appears as predicted frustums that are oriented correctly but bunched together or shifted relative to GT.

3.3 Extra Credit: number of input views at evaluation

Question: how does the number of input views change accuracy, runtime and memory for a model trained with 8 views, without retraining?

Settings: scripts/vggt_view_count.py evaluates the same best.pt on the validation split with 2, 4, 8 and 16 views, using the 3.2 protocol (first 32 prepared frames, seed 825). The view sets are nested: each scene uses frame 0 plus the first entries of one fixed permutation, so going from 4 to 8 views only adds frames. Runtime is the median of 20 forward passes on one scene (bf16, model only); memory is the peak allocation of that pass including the 0.6 GB of weights.

Views AbsRel δ1 Rot AUC@30 Trans AUC@30 Joint AUC@30 ms / scene ms / view Peak GB
2 0.122 0.841 0.687 0.206 0.206 21.9 10.96 0.70
4 0.133 0.846 0.700 0.180 0.162 22.7 5.68 0.72
8 (trained) 0.152 0.804 0.717 0.188 0.165 22.4 2.80 0.78
16 0.179 0.781 0.677 0.165 0.139 27.6 1.73 0.88

Accuracy, time and memory vs number of views

Conclusions. Rotation accuracy peaks at the training view count and drops at 16 views, where the global attention sees a sequence length it was never trained on. Depth does not improve with more views; it gets worse, AbsRel rising from 0.12 at 2 views to 0.18 at 16. Three plausible causes. Global attention over a sequence length other than the 8 x 261 tokens seen in training shifts the feature statistics for every token, including the patch tokens the depth head reads. The scene scale is recomputed from the selected views, so the normalized depth targets themselves change with the view count. And at 2 views half the evaluated pixels belong to frame 0, the reference camera with its own learned token, which the model handles best. Either way, extra views give no accuracy benefit at test time for this model: the task is close to per-view depth with cross-view consistency, and extra context does not compensate for the train/test mismatch. Cost grows slowly: forward time is flat at about 22 ms up to 8 views and 28 ms at 16, and peak memory rises from 0.70 to 0.88 GB. At these sizes the frozen DINOv2 encoder and fixed overheads dominate, and the quadratic global attention (16 x 261 = 4176 tokens) is still cheap on this GPU; per-view cost falls from 11 ms to 1.7 ms as views are batched.

Limitations: a single checkpoint trained at 8 views, so the comparison mixes "more information" with "train/test view-count mismatch"; separating them would need retraining at each view count. The depth metrics are not over identical pixels or identical normalization across rows. Timing uses a batch of one scene, which understates the GPU's throughput at higher view counts.