Q1
1.2 Unprojection visualization
1.3 Canonicalization visualization
1.7 Final Check
Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00010235906665911898,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Canonicalization: Every pose is multiplied by the inverse of camera 0's pose, so camera 0 becomes the identity at the origin. Then translations, depths, and points are divided by the distance from camera 0 to the centroid of the valid points, which puts the centroid at distance 1. This removes the arbitrary world origin, orientation, and metric scale of each scan, so all scenes share one normalized frame. The scale is stored separately so evaluation can undo it.
World-space ray directions: For pixel (u, v) the camera-frame direction is K^-1 [u, v, 1]. Rotating it by the camera-to-world rotation and normalizing gives a unit direction in the shared frame. The ray origin is the camera center. These define the 3D line each pixel looks along, which lets the model relate pixels across views.
Plücker moment: A line has no natural starting point, so Plücker coordinates [d, m] describe it without one: d is the unit direction and m = o × d is the moment. Moving the origin along the line changes o by a multiple of d, which leaves m unchanged. The moment's length is the line's distance from the world origin and its direction is the normal of the plane through the line and the origin. The 6 values per pixel are the camera conditioning for the Mini-LVSM tokens.
Q2
2.1
overfit PSNR: 30.929217020670574
2.2 Training and Evaluation
Training settings: default configs/lvsm.json. Patch size 8, width 384, 8 transformer layers, head dim 64, MLP ratio 4, QK-norm. 8 views per sample at 224 px (sampling range 32), 2 source views and 6 targets, batch size 4, 100 epochs (5600 steps), AdamW with lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps, cosine decay, grad clip 1.0, bf16 autocast with the MSE loss in float32. Training took 17.5 minutes on one RTX 5090.
Training curves (train and validation, per epoch):
Final epoch: train PSNR 15.60 dB, validation PSNR 16.13 dB, SSIM 0.560. Validation is slightly above train because training batches use random view sampling and the train number is averaged while the weights are still changing within the epoch; there is no overfitting, and both curves are still rising at epoch 100, so the model is underfit.
Validation ablations (python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none):
| Source views | Conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.558 |
| 1 | shuffled | 11.80 | 0.487 |
| 1 | none | 12.45 | 0.526 |
| 2 | correct | 16.13 | 0.560 |
| 2 | shuffled | 11.49 | 0.472 |
| 2 | none | 12.43 | 0.525 |
| 4 | correct | 16.87 | 0.575 |
| 4 | shuffled | 11.58 | 0.472 |
| 4 | none | 12.44 | 0.536 |
Visualizations. Columns: source views, ground truth, prediction. One row per target view.
Training visualization (validation batch at epoch 95, S = 2):
Evaluation, number of source views (correct rays), S = 1 / 2 / 4:
Evaluation, ray conditioning at S = 2, shuffled / none:
Number of source views: quality improves monotonically with more sources (15.19, 16.13, 16.87 dB), even though the model was trained with 2. Extra views cover more of the target region and give the attention more evidence for each query ray, with diminishing returns from 2 to 4. The model generalizes to a sequence length it never saw in training because the token layout is the same, just longer.
Ray conditioning: the target rays are essential. Shuffled rays drop PSNR by about 4.6 dB and zero rays by about 3.7 dB, a larger effect than the source count. Shuffled is worse than none: with wrong rays the model confidently renders the wrong viewpoint, while with no rays it falls back to a blurry average of the sources, which scores higher under MSE. The gap shows that the model is not simply copying sources but reading the Plücker rays to decide which content lands in each patch.
Predictions are blocky and low-detail at every setting. With 8x8 patches regressed directly to pixels and only 5600 training steps, the model captures layout and color but not texture, consistent with the reference numbers (about 15 / 16 / 17 dB for S = 1 / 2 / 4).
2.3 Extra Credit: model capacity (patch size)
Question: how does the patch size affect quality and training cost? Doubling the patch from 8 to 16 cuts the token count per sequence by 4x (2352 to 588) and the attention cost by about 16x, but each token must then regress a 16x16x3 = 768-value pixel block instead of 192.
Settings: configs/lvsm_patch16.json is the 2.2 config with only patch_size changed from 8 to 16. Same data, width 384, 8 layers, 2 source views, batch 4, 100 epochs, lr 2e-4, bf16. Both runs on one RTX 5090 with 4 data-loader workers. Parameters: 14.6M (patch 8) vs 16.0M (patch 16); the difference is the larger input and output projections.
Results (validation, best.pt):
| Patch 8 (baseline) | Patch 16 | |
|---|---|---|
| PSNR / SSIM, S=1 correct | 15.19 / 0.558 | 15.02 / 0.554 |
| PSNR / SSIM, S=2 correct | 16.13 / 0.560 | 15.89 / 0.555 |
| PSNR / SSIM, S=4 correct | 16.87 / 0.575 | 16.52 / 0.567 |
| PSNR, S=2 shuffled / none | 11.49 / 12.43 | 11.49 / 12.37 |
| Tokens per sequence | 2352 | 588 |
| Median time per epoch | 11.2 s | 7.3 s |
| Total training time | 17.5 min | 11.3 min |
Validation curves of the two runs:
Evaluation at S = 2 with correct rays, patch 8 (top) vs patch 16 (bottom). Columns: sources, ground truth, prediction.