Multi-view transformers: ScanNet++ preprocessing, Mini-LVSM and Mini-VGGT.
Runs used Python 3.10, PyTorch 2.4.1 with CUDA 12.1 and one NVIDIA A10G. The dataset contains 224 training, 28 validation and 28 test scenes. All reported held-out evaluations use every scene in their split.
Intrinsics follow the half-pixel resize convention. Depth is optical-axis z-depth. I unproject with K inverse, then rotate and translate into world coordinates. Canonicalization applies the inverse of the first sampled camera pose and divides translations, depth and points by the distance to the valid-point centroid. The first camera is identity and the centroid distance is one; the centroid itself is not subtracted.
A world-space ray starts at the camera center. Its direction is K inverse times the homogeneous pixel coordinate, rotated by the camera-to-world rotation and normalized. The Plücker encoding is [d, m], with m = o × d. The moment encodes the ray's offset from the origin and is perpendicular to its direction.
Unprojection, training scene 1, meters.
Canonicalization, same scene 1. Camera 0 and centroid distance are shown in the sidebar.
Unprojection, training scene 2, meters.
Canonicalization, same scene 2. Camera 0 and centroid distance are shown in the sidebar.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2,000 steps on one fixed training clip: final PSNR 37.290 dB, MSE 0.000187. Batch size 1, learning rate 0.001, 50 warmup steps. Two source views and six targets; target RGB never enters the model.
Columns: source 1, source 2, ground truth, prediction. Each row is a target.
Baseline: 16px patches, width 192, four transformer blocks, three attention heads, MLP ratio 4, Q/K normalization. Eight views at 224px, two sources, sampling range 32, batch size 4, 100 epochs over all 224 training scenes (56 steps/epoch), seed 825. AdamW: learning rate 0.0002, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps, cosine decay, gradient clipping at 1, BF16 training. The best validation PSNR checkpoint is evaluated in FP32.
LVSM training and validation curves, including the controlled capacity comparison.
| Sources | Target rays | PSNR (dB) | SSIM | Target images |
|---|---|---|---|---|
| 1 | correct | 14.7783 | 0.53982 | 196 |
| 1 | shuffled | 11.8422 | 0.48810 | 196 |
| 1 | none | 12.3961 | 0.46342 | 196 |
| 2 | correct | 15.4960 | 0.53596 | 168 |
| 2 | shuffled | 11.5703 | 0.47366 | 168 |
| 2 | none | 12.3230 | 0.45974 | 168 |
| 4 | correct | 15.9845 | 0.54812 | 112 |
| 4 | shuffled | 11.7598 | 0.47941 | 112 |
| 4 | none | 12.3169 | 0.46376 | 112 |
With correct rays, PSNR for 1, 2 and 4 sources is 14.778, 15.496 and 15.985 dB. At two sources, correct-minus-shuffled and correct-minus-zero PSNR differences are +3.926 and +3.173 dB. Additional sources provide more appearance coverage, while target rays specify the requested camera. The source-count comparison also changes which views remain targets; only two-source inputs were seen during training. The predictions retain coarse color/layout but visibly lose furniture edges and fine texture. Here shuffled rays score below zero rays: the query requests a different camera. With zero rays, all target-query sequences in a clip are identical, so the model produces one common prediction. Zero rays are also outside the training distribution.
Training validation visualization: val_epoch090.png. Columns: two sources, GT, prediction.
Validation: 1 source(s), correct target rays. Columns: sources, ground truth, prediction.
Validation: 1 source(s), shuffled target rays. Columns: sources, ground truth, prediction.
Validation: 1 source(s), none target rays. Columns: sources, ground truth, prediction.
Validation: 2 source(s), correct target rays. Columns: sources, ground truth, prediction.
Validation: 2 source(s), shuffled target rays. Columns: sources, ground truth, prediction.
Validation: 2 source(s), none target rays. Columns: sources, ground truth, prediction.
Validation: 4 source(s), correct target rays. Columns: sources, ground truth, prediction.
Validation: 4 source(s), shuffled target rays. Columns: sources, ground truth, prediction.
Validation: 4 source(s), none target rays. Columns: sources, ground truth, prediction.
Question: do smaller patches improve the very soft baseline predictions enough to justify their cost? The 16px baseline and 8px variant use the same width, four blocks, data, seed, source count, optimizer and 100-epoch budget. Both use their best validation checkpoint. The table uses two sources and correct target rays on all 28 validation scenes.
| Patch size | Parameters | PSNR | SSIM | Forward (ms) | Peak allocated (MiB) | Train + val (min) |
|---|---|---|---|---|---|---|
| 16px patches | 2664832 | 15.4960 | 0.53596 | 4.07 | 79.2 | 66.5 |
| 8px patches | 2000704 | 15.7466 | 0.55318 | 26.41 | 212.6 | 67.1 |
8px versus 16px patches changes PSNR by +0.251 dB and SSIM by +0.0172. The forward-time ratio is 6.49. The quality gain is modest relative to the latency increase. The finer patch grid reduces block size, but furniture edges and texture remain blurred. Halving patch size gives four times as many tokens per view and sixteen times as many full-attention token pairs. The input/output linear layers have fewer parameters, and each output token covers fewer pixels. This can reduce the per-patch information bottleneck, but MSE averaging and limited training can still cause blur. This is one seed and a fixed training budget, so it does not establish a universal patch-size ranking. Timing uses batch 1, two sources, six targets, FP32, three warmups and the median of ten forwards, excluding data loading and transfer. Peak allocated memory includes the model and input batch. Training wall times include I/O and validation, and overlap with independent VGGT runs; use the isolated forward measurements to compare model cost.
8px patch model, same validation clip and source/GT/prediction layout as the 16px baseline.
One fixed training clip, eight views, 40 epochs × 25 steps = 1,000 steps, learning rate 0.001 with cosine decay. The pretrained DINOv2 encoder stays frozen.
Fixed-clip total, camera and depth losses.
Fixed training clip: gt.
Fixed training clip: prediction.
Frozen pretrained DINOv2 ViT-S/14; width 192; six heads; four frame/global pairs (eight aggregation blocks); the starter four-block camera head and RGB-refined depth head. Eight views, sampling range 32, 224px, batch size 4, 150 epochs over all 224 training scenes, seed 825. AdamW learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine minimum ratio 0.05, gradient clipping 1, BF16 forwards. Loss weights: camera 5, depth 1. The optional point head is implemented but disabled in these runs.
Full training and validation total, camera and depth losses.
Both splits use the best validation-loss checkpoint, eight views from the first 32 prepared frames, 224px and seed 825. No batch limit, GT alignment or predicted-confidence pixel filtering is used. AUC values are in [0, 1].
| Split | Scenes | Depth AbsRel | Depth δ1 | Rotation AUC 5° / 15° / 30° | Translation AUC 5° / 15° / 30° | Joint pose AUC 5° / 15° / 30° |
|---|---|---|---|---|---|---|
| Validation | 28 | 0.17434 | 0.75078 | 0.1196 / 0.2777 / 0.4412 | 0.0094 / 0.0382 / 0.0939 | 0.0018 / 0.0239 / 0.0777 |
| Test | 28 | 0.19955 | 0.69270 | 0.1247 / 0.3047 / 0.4941 | 0.0059 / 0.0330 / 0.0974 | 0.0010 / 0.0196 / 0.0773 |
Test depth AbsRel is 0.200, versus 0.174 on validation; test δ1 is 0.693. Test rotation, translation and joint AUC at 30° are 0.494, 0.097 and 0.077. Translation-direction errors dominate the joint pose result. In the dining-room example, chair/table depth boundaries are smoothed; in the corridor, reconstructed walls bend and overlap. Predicted camera centers are more tightly clustered than GT, indicating underestimated motion. GT and predictions share coordinates and axis ranges without alignment. Small baselines make direction errors sensitive to position offsets. The compact model and frozen features may also limit pose generalization; these experiments do not isolate those causes.
Held-out scene 1: gt.
Held-out scene 1: prediction.
Held-out scene 2: gt.
Held-out scene 2: prediction.
Held-out scene 3: gt.
Held-out scene 3: prediction.
Held-out scene 4: gt.
Held-out scene 4: prediction.
Question: how do input views affect accuracy, latency and memory for a model trained with eight views? The same checkpoint is evaluated with 2, 4, 8 and 16 views on all 28 validation scenes. The seed, 32-frame prefix, resolution and weights stay fixed. View selections are nested. The shared loader recomputes canonical scale after view selection, as specified by the data contract.
| Views | AbsRel | δ1 | Joint AUC 5° / 15° / 30° | Forward (ms) | Peak allocated (MiB) |
|---|---|---|---|---|---|
| 2 | 0.13608 | 0.80671 | 0.0000 / 0.0190 / 0.0869 | 27.16 | 175.3 |
| 4 | 0.16633 | 0.75705 | 0.0000 / 0.0242 / 0.0815 | 27.40 | 193.9 |
| 8 | 0.17434 | 0.75078 | 0.0018 / 0.0239 / 0.0777 | 27.69 | 231.6 |
| 16 | 0.20985 | 0.67105 | 0.0013 / 0.0231 / 0.0725 | 28.34 | 306.7 |
Compared with eight views, two views change AbsRel by -0.0383; 16 views change it by +0.0355. Measured latency spans 27.16–28.34 ms, while peak memory rises from 175.3 to 306.7 MiB. Global attention has quadratic token interactions; fixed Python/kernel-launch overhead is a possible explanation for the nearly flat latency, but this benchmark does not isolate that cause. More views did not improve the all-view depth score here. View selection and canonical scale also change the evaluation targets, and non-eight-view inputs differ from the training distribution. Pairwise AUC compares different camera pairs across view counts, so it is not a matched-pair estimate. Timing includes the encoder and prediction heads: BF16, batch 1, three warmups, median of ten forwards; data loading, transfer, losses and visualization are excluded. Memory is peak allocated CUDA memory, including model and inputs.
Same validation scene reconstructed with 2 input views.
Same validation scene reconstructed with 16 input views.