1. Multi-View Data Preprocessing
Unprojection
Canonicalization
Camera 0 (red frustum) is at the origin, first camera center = (0, 0, 0), valid-point centroid distance = 1.000.
check_data.py
| Sample | Valid depth | Point reproj. (px) | Ray reproj. (px) | Centroid dist. |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.978 | 1.1e-04 | 1.1e-04 | 1.0000 |
| 5f99900f09_frame_002820 | 0.990 | 1.2e-04 | 1.1e-04 | 1.0000 |
| 47b37eb6f9_frame_002720 | 0.961 | 9.9e-05 | 8.9e-05 | 1.0000 |
| 4c5c60fa76_frame_006690 | 0.752 | 1.1e-04 | 1.0e-04 | 1.0000 |
Full output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Explanations
- Canonicalization. Each sample is expressed in its first camera's frame (camera 0 = identity), and translations, depth and points are divided by the distance from that camera to the centroid of the valid points, so all scenes share one coordinate frame and scale.
- World-space ray directions.
d = normalize(R K-1[u, v, 1]T)is the unit direction of a pixel's viewing ray in the canonical frame; the ray origin is the camera center. - Plücker moment.
m = o x dis perpendicular to the plane spanned byoandd, and its length is the distance from the world origin to the ray. It does not change whenoslides along the ray, so[d, m]is a unique description of the line.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
final PSNR 30.96 dB
2.2 Training and Evaluation
Settings: default configs/lvsm.json: 8 views, S = 2 source views, sampling range 32, 224 × 224, learning rate 2e-4, batch size 4, MSE loss, except 300 epochs instead of the default 100.
Ablations
Validation
Test
| Source views | Target rays | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.76 | 0.5833 |
| 1 | shuffled | 11.65 | 0.4824 |
| 1 | none | 12.45 | 0.5335 |
| 2 | correct | 16.97 | 0.5915 |
| 2 | shuffled | 11.25 | 0.4616 |
| 2 | none | 12.40 | 0.5306 |
| 4 | correct | 18.10 | 0.6108 |
| 4 | shuffled | 11.29 | 0.4576 |
| 4 | none | 12.44 | 0.5411 |
| Source views | Target rays | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 16.15 | 0.5628 |
| 1 | shuffled | 12.25 | 0.4667 |
| 1 | none | 12.77 | 0.5144 |
| 2 | correct | 17.08 | 0.5749 |
| 2 | shuffled | 12.06 | 0.4557 |
| 2 | none | 12.72 | 0.5158 |
| 4 | correct | 18.23 | 0.5808 |
| 4 | shuffled | 11.57 | 0.4327 |
| 4 | none | 12.71 | 0.5108 |
Visualizations
Training, epoch 0
Training, epoch 295
Evaluation, S = 1
Evaluation, S = 2
Evaluation, S = 4
S = 2, shuffled rays
S = 2, no rays
Discussion
- Number of source views. Validation PSNR rises as number of source views increases, since more sources cover more of the target view.
- Ray conditioning. Shuffled rays are worse than zero rays, and both are far below correct rays. So the model relies on the target rays to decide what to render. With zero rays, PSNR does not depend on S.
2.3 Bonus: Loss Function
Question. What happens to PSNR, SSIM and visual sharpness when an L1, SSIM or perceptual term is added to the MSE loss, and is the effect larger than the run-to-run variation?
Setup. Every model uses the Q2.2 configuration and differs only in the loss and, for the repeats, the training seed. Runs: MSE; MSE + 1.0 × L1; MSE + w × (1 − SSIM) with w = 1.0, 0.3, 0.1; MSE + 0.1 × perceptual loss (mean L1 distance between frozen ImageNet VGG16 features at relu1_2, relu2_2, relu3_3, relu4_3). MSE, L1 and SSIM with w = 0.1 were repeated with seeds 825, 1 and 2. Sharpness ratio is mean absolute horizontal + vertical image gradient of the prediction divided by that of the ground truth (smaller = blurrier).
Loss weights (seed 825)
Validation
Test
| Loss | PSNR (dB) | SSIM | Sharpness ratio |
|---|---|---|---|
| MSE (baseline) | 16.14 | 0.5601 | 0.379 |
| MSE + 1.0 × L1 | 16.28 | 0.5634 | 0.403 |
| MSE + 1.0 × (1 − SSIM) | 14.87 | 0.5751 | 0.127 |
| MSE + 0.3 × (1 − SSIM) | 15.42 | 0.5803 | 0.163 |
| MSE + 0.1 × (1 − SSIM) | 15.55 | 0.5778 | 0.194 |
| MSE + 0.1 × VGG16 perceptual | 14.70 | 0.5615 | 0.321 |
| Loss | PSNR (dB) | SSIM | Sharpness ratio |
|---|---|---|---|
| MSE (baseline) | 16.26 | 0.5431 | 0.345 |
| MSE + 1.0 × L1 | 16.37 | 0.5451 | 0.372 |
| MSE + 1.0 × (1 − SSIM) | 14.88 | 0.5551 | 0.119 |
| MSE + 0.3 × (1 − SSIM) | 15.45 | 0.5615 | 0.151 |
| MSE + 0.1 × (1 − SSIM) | 15.61 | 0.5594 | 0.176 |
| MSE + 0.1 × VGG16 perceptual | 14.74 | 0.5429 | 0.290 |
Three seeds (mean ± std)
Validation
Test
| Loss | PSNR (dB) | SSIM | Sharpness ratio |
|---|---|---|---|
| MSE (baseline) | 16.13 ± 0.00 | 0.5589 ± 0.0012 | 0.380 ± 0.003 |
| MSE + 1.0 × L1 | 16.28 ± 0.04 | 0.5635 ± 0.0001 | 0.393 ± 0.009 |
| MSE + 0.1 × (1 − SSIM) | 15.77 ± 0.19 | 0.5822 ± 0.0038 | 0.201 ± 0.007 |
| Loss | PSNR (dB) | SSIM | Sharpness ratio |
|---|---|---|---|
| MSE (baseline) | 16.22 ± 0.03 | 0.5411 ± 0.0018 | 0.349 ± 0.004 |
| MSE + 1.0 × L1 | 16.36 ± 0.03 | 0.5446 ± 0.0008 | 0.361 ± 0.010 |
| MSE + 0.1 × (1 − SSIM) | 15.79 ± 0.17 | 0.5635 ± 0.0036 | 0.184 ± 0.007 |
Columns: ground truth, then the six seed-825 models
Results.
- L1. A small gain in PSNR, SSIM and sharpness, present for every seed on both splits and larger than the seed variation. The images look almost the same.
- SSIM term. It raises SSIM but lowers PSNR and sharpness, and a smaller weight reduces the drop. The predictions become smoother. These runs also vary more between seeds, but every run is below the MSE run with the same seed.
- Perceptual loss. Lower PSNR than MSE and similar SSIM; the predictions are smoother with a color shift, some high frequency details are lost.
Explanation: L1 is less dominated by a few large errors than MSE, a plausible reason for its small, consistent gain. SSIM compares local mean, variance and covariance, so a smooth prediction with the right brightness layout already scores well. Since the model cannot predict fine texture, the SSIM term pushes it toward smooth outputs, raising SSIM and lowering PSNR and sharpness. A higher SSIM here is therefore not better visual quality. The MSE and L1 predictions show the 8 × 8 patch grid, the patch edges also create image gradients, so their sharpness ratio partly counts block edges instead of real detail. The perceptual loss uses pretrained VGG features to compare textures that this small, undertrained model cannot reproduce, which may explain why there is no gain.
Limitations: One weight and one seed for the perceptual loss (a smaller weight or longer training might behave differently), and all comparisons use 100 epochs and S = 2.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
Ground truth
Predicted camera + depth
3.2 Training and Evaluation
Settings: default configs/vggt.json: 8 views, sampling range 32, 224 × 224, learning rate 6e-4, batch size 16, 1000 epochs, loss = 5 × camera + 1 × depth.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel (lower is better) | 0.141 | 0.160 |
| Depth δ1 | 0.840 | 0.797 |
| Rotation AUC@5° | 0.185 | 0.201 |
| Rotation AUC@15° | 0.505 | 0.496 |
| Rotation AUC@30° | 0.690 | 0.697 |
| Translation AUC@5° | 0.018 | 0.018 |
| Translation AUC@15° | 0.090 | 0.087 |
| Translation AUC@30° | 0.208 | 0.198 |
| Joint pose AUC@5° | 0.007 | 0.005 |
| Joint pose AUC@15° | 0.071 | 0.056 |
| Joint pose AUC@30° | 0.186 | 0.171 |
Validation scene 1: 09c1414f1b
Ground truth
Predicted camera + depth
Validation scene 2: 21d970d8de
Ground truth
Predicted camera + depth
Validation scene 3: 280b83fcf3
Ground truth
Predicted camera + depth
Validation scene 4: 2b1dc6d6a5
Ground truth
Predicted camera + depth
Discussion
- Depth is reasonably accurate. On unseen scenes the depth is off by about 15% on average. The predicted depth maps are smooth because the model works on patches, thin objects and edges get blurred. The point clouds show the rough room layout and are noisier than GT.
- Rotation is fair, but translation is poor. The rotation error is about 11° and roughly half of the camera pairs are within 15°. The translation direction is much worse: about 40° error and translation / joint AUC@15° below 0.1. The views are sampled from a short stretch of video, so the cameras barely move and the translation is hard to estimate.
- Generalization gap. The same model fits a single clip almost perfectly. However, there are only 224 training scenes, and the validation loss was still going down slowly at the end. Longer training or using more data would probably help.
3.3 Bonus: Number of Input Views
Question. The model was trained with 8 views. How do accuracy, runtime and memory change when the number of input views changes at evaluation?
Setup. Same best.pt on the validation split with V ∈ {2, 4, 8, 16, 32} views taken from the first 32 prepared frames. Time is wall-clock per scene including data loading and memory is peak allocated GPU memory.
| Views | AbsRel | δ1 | Rot. AUC@15° | Trans. AUC@15° | Joint AUC@15° | s / scene | Peak GB |
|---|---|---|---|---|---|---|---|
| 2 | 0.116 | 0.853 | 0.543 | 0.169 | 0.169 | 0.08 | 0.70 |
| 4 | 0.126 | 0.861 | 0.498 | 0.096 | 0.070 | 0.06 | 0.75 |
| 8 | 0.141 | 0.841 | 0.505 | 0.089 | 0.070 | 0.08 | 0.84 |
| 16 | 0.158 | 0.814 | 0.452 | 0.075 | 0.051 | 0.27 | 0.98 |
| 32 | 0.164 | 0.784 | 0.451 | 0.070 | 0.048 | 0.55 | 1.29 |
Results. More views at test time did not help this model. Depth AbsRel grows and joint pose AUC@15° drops. δ1 peaks at V = 4. Peak memory rises from 0.70 to 1.29 GB. Time per scene barely changes at first, then rises to 0.55s, since global attention cost grows with the total number of tokens.
Explanation. The model was trained with 8 views only. With 16 or 32 views, the global attention and the camera head get inputs they never saw in training, so the accuracy drops.
Limitations. The comparison across V is not exact, because the scene scale is recomputed from the chosen views and the sampled frames differ. The AUC also covers a different number of camera pairs for each V. With V = 2 there is only one pair per scene. The time includes data loading. This is one checkpoint and one seed, and the model was not retrained. Training with different numbers of views would show whether the drop comes from more views or from the mismatch with training.



























