1. Multi-View Data Preprocessing
Canonicalization: Express cameras and points in the first sampled camera's frame using
T0⁻¹ Ti. Divide translations, depth and points by s = ‖mean(pvalid)‖, without subtracting the centroid.
Camera 0 is at the origin and the valid-point centroid has distance one.
World-space rays: The origin o is the camera center. Rotate K⁻¹[u,v,1]ᵀ into world coordinates
and normalize it to obtain direction d. Depth unprojection uses optical-axis z-depth, not distance along a unit ray.
Plücker moment: Each ray is [d, o × d]. The moment describes the line's offset and is unchanged
when the origin moves along the ray.
Unprojection and canonicalization


check_data.py output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM
Q2.1 — Overfitting
After 2,000 steps on one fixed batch: 30.9210 dB PSNR.

Q2.2 — Training and evaluation
Settings: 100 epochs on 224 training scenes; 8 views, 2 sources, batch size 4, 224px, seed 825. Patch size 8, width 384, 8 transformer blocks. AdamW, learning rate 0.0002, weight decay 0.05, 200 warmup steps, cosine decay, gradient clipping 1.0 and bfloat16 training with float32 MSE loss. Validation uses 28 scenes and eight views from the first 32 prepared frames. Best validation PSNR is 16.1162 dB at epoch 100.


Validation ablations
All settings use the same best checkpoint. Correct uses true target rays; shuffled uses another target view's rays; none uses zero target rays. Higher PSNR and SSIM are better.
| Sources | Target rays | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.1967 | 0.55785 |
| 1 | shuffled | 11.8019 | 0.48694 |
| 1 | none | 12.4683 | 0.52908 |
| 2 | correct | 16.1162 | 0.55987 |
| 2 | shuffled | 11.4815 | 0.47194 |
| 2 | none | 12.4388 | 0.52679 |
| 4 | correct | 16.8572 | 0.57494 |
| 4 | shuffled | 11.5893 | 0.47337 |
| 4 | none | 12.4562 | 0.53640 |
With correct rays, PSNR increases from 15.20 to 16.12 to 16.86 dB as source count increases. At two sources, shuffling rays reduces PSNR from 16.12 to 11.48 dB; removing them gives 12.44 dB. This indicates that the model uses camera conditioning. More sources provide additional scene information, though source-count comparisons also change the target set and test source counts different from training. Predictions capture broad colors and layout but remain blurred, especially at object boundaries and fine textures. These single-seed results do not establish robust geometric extrapolation.
1 source views: evaluation images



2 source views: evaluation images



4 source views: evaluation images



3. Multi-View Geometry with Mini-VGGT
Q3.1 — Overfitting
One fixed training clip, 40 epochs × 25 steps, learning rate 0.001. Final depth AbsRel 0.03565, δ1 0.95710, and joint pose AUC@30° 0.99762.



Q3.2 — Training and evaluation
Settings: frozen DINOv2 ViT-B/14, eight frame/global attention pairs, width 384, six heads; 1000 epochs on 224 training scenes, batch size 16, 8 views, sampling range 32, 224px, seed 825. AdamW, learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine decay to 5% of peak learning rate, gradient clipping 1.0 and bfloat16 attention. Best checkpoint: epoch 850, selected by validation loss.

Validation and test each use all 28 scenes, eight views from the first 32 prepared frames, 224px and seed 825, with no batch limit. Lower AbsRel is better; higher δ1 and AUC are better.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.14581 | 0.15223 |
| Depth δ1 | 0.82713 | 0.82036 |
| Rotation AUC@5° | 0.16964 | 0.20102 |
| Rotation AUC@15° | 0.50374 | 0.54685 |
| Rotation AUC@30° | 0.70523 | 0.74949 |
| Translation AUC@5° | 0.02194 | 0.02066 |
| Translation AUC@15° | 0.09252 | 0.09379 |
| Translation AUC@30° | 0.19698 | 0.22976 |
| Joint pose AUC@5° | 0.00281 | 0.00408 |
| Joint pose AUC@15° | 0.06718 | 0.06463 |
| Joint pose AUC@30° | 0.17666 | 0.20557 |
Held-out geometry and errors
Predicted depth captures broad scene structure, but camera errors can misalign surfaces across views. Among these four scenes, 280b83fcf3 has the highest depth AbsRel and mean rotation error. Translation errors remain substantial, consistent with the lower translation and joint pose AUCs. Depth and relative-pose accuracy measure different aspects of reconstruction; accurate per-view depth does not guarantee aligned point clouds.
Scene 1: 09c1414f1b_frame_000000
Depth AbsRel 0.0873, δ1 0.9412; mean relative rotation error 4.68°, translation-direction error 27.33°; joint pose AUC@30° 0.3655.


Scene 2: 21d970d8de_frame_000000
Depth AbsRel 0.0593, δ1 0.9648; mean relative rotation error 3.84°, translation-direction error 44.83°; joint pose AUC@30° 0.0679.


Scene 3: 280b83fcf3_frame_000000
Depth AbsRel 0.1064, δ1 0.8924; mean relative rotation error 13.47°, translation-direction error 42.03°; joint pose AUC@30° 0.2012.


Scene 4: 2b1dc6d6a5_frame_000000
Depth AbsRel 0.0789, δ1 0.9713; mean relative rotation error 6.72°, translation-direction error 10.31°; joint pose AUC@30° 0.5881.


Q3.3 — Number of input views
Question: how do 2, 4 and 8 input views affect geometry accuracy, inference time and GPU memory?
Settings: the same epoch-850 checkpoint, trained with eight views, is evaluated on all 28 validation scenes. Inputs are nested subsets of eight frames sampled from the first 32 frames (seed 825). Each setting scores the same two views and camera pair, with normalization fixed. No retraining or alignment is used. Batch size 1, 224px, NVIDIA L4, bfloat16.
Latency is the median synchronized forward time after three warmups and ten repetitions per scene/setting, excluding data loading, transfer and metrics. Memory is peak allocated tensor memory, including the model, rather than total GPU process memory.
| Input views | AbsRel ↓ | δ1 ↑ | Rotation AUC@30° ↑ | Translation AUC@30° ↑ | Joint pose AUC@30° ↑ | Latency (ms) | Peak memory (MiB) |
|---|---|---|---|---|---|---|---|
| 2 | 0.19806 | 0.65855 | 0.67976 | 0.21310 | 0.20238 | 27.31 | 590.2 |
| 4 | 0.15341 | 0.79786 | 0.70357 | 0.19286 | 0.17738 | 27.09 | 606.5 |
| 8 | 0.14228 | 0.82214 | 0.69405 | 0.22381 | 0.20833 | 27.47 | 641.5 |


Conclusion: eight views reduce AbsRel from 0.1981 to 0.1423, improving depth in 21/28 scenes. Pose accuracy is not monotonic. Latency remains near 27 ms with overlapping timing ranges, while memory rises from 590 to 641 MiB. Extra context can help resolve scene structure, but redundant views and occlusion can limit pose gains. Shared weights and fixed dispatch costs may explain the modest memory increase and similar latency; thin structures remain blurred.
Limitations: one checkpoint, one frame selection per scene, and only two scored views/one camera pair. Fewer views differ from training, and no models were retrained. Results do not establish training-view-count effects; timing is specific to this GPU and batch size.