1. Multi-View Data Preprocessing
Depth Unprojection
Depth from all eight sampled views is unprojected with the resized intrinsics and the original camera-to-world poses, so the points are in the dataset's metric world frame.

*Figure 1: Unprojection for scene 036bce3393. (Red frustum: camera0, blue frustums: the other seven sampled views.)




Camera and Depth Canonicalization
All cameras and points are expressed in camera 0's frame, then divided by the distance of the valid-point centroid, so every sample shares the same frame and scale. The first camera center and valid-point centroid distance are described in the right pane of each screenshot.

*Figure 3: Canonicalization for scene 036bce3393. All cameras and points are expressed in camera 0's frame. (Red frustum: camera0, blue frustums: the other seven sampled views.)




Canonicalization: Every pose is multiplied by the inverse of camera 0's pose, so camera 0 becomes the identity and the other cameras are expressed relative to it. Depth is unprojected in this frame, and translations, depth and points are all divided by one scene scale, which is the distance from camera 0 to the centroid of the valid points, which ends up at distance 1.
World-space ray directions: Each image pixel corresponds to a ray starting at the camera center and passing through that pixel. The world-space ray direction is this direction after transforming it from the camera's local coordinate system into the shared 3D world coordinate. Using world space lets the model compare rays from different camera views consistently.
Plücker moment: Plücker moment is a 6D encoding (d, m) which uniquely describes a 3D line relative to the world origin. d is the unit direction and m is the moment m = o x d where o is the origin. The moment is invariant to sliding the origin along the line.
Final Check
All four training samples pass. Reprojection errors are much below the 0.01 px threshold and the valid-point centroid stays at distance 1 from camera 0.
Full output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010810750973178074,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.0001155954014393501,
"max_ray_reprojection_error_px": 0.00010235906665911898,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 8.28983829705976e-05,
"max_ray_reprojection_error_px": 7.938443741295487e-05,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 9.5367431640625e-05,
"max_ray_reprojection_error_px": 8.996945689432323e-05,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
I implemented the relevant code and confirmed the implemented model can memorize one fixed batch. The final overfit PSNR and the val_epoch000.png visualization are as follows:

2.2 Training and Evaluation
I implemented the relevant code and trained on the full training split with the training settings below. I show the training curves in Fig. 6, validation PSNR/SSIM for each ablation setting in Fig. 7, and the training and evaluation visualization
Training settings. I used the settings from configs/lvsm.json (shown below) without changes.
| Setting | Value |
|---|---|
| Epochs | 100 |
| Source / target views | 1,2,4 / 6 |
| Batch size | 4 |
| Optimizer | AdamW |
| Learning rate | 0.0002 |
| Weight decay | 0.05 |
| Betas | (0.9 0.95) |
| Loss | MSE on target pixels |
| Warmup steps | 200 |
| Gradient clipping | 1.0 |

| Source views | Conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | correct | 15.18 | 0.558 |
| 1 | shuffled | 11.81 | 0.488 |
| 1 | none | 12.47 | 0.530 |
| 2 | correct | 16.13 | 0.561 |
| 2 | shuffled | 11.50 | 0.473 |
| 2 | none | 12.45 | 0.527 |
| 4 | correct | 16.86 | 0.575 |
| 4 | shuffled | 11.60 | 0.474 |
| 4 | none | 12.46 | 0.536 |






Effect of the number of source views. The validation PSNR/SSIM in the Table 2 shows more source views improve both PSNR and SSIM. PSNR rises from 15.18 dB with one view to 16.86 dB with four, and SSIM increases from 0.558 to 0.575. Extra views cover more of the target view and give the model parallax to triangulate from, where as a single source forces it to guess the geometry and give a blurrier result. The gain carries over to four sources even though training used two because all sources share the same embedding and projection.
Effect of ray conditioning. The target rays are the only information about the target viewpoint, and the model depends on them completely. With shuffled rays, PSNR drops by about 5dB at every numbers of source views and render an image of the wrong view (Fig. 9, middle). With zero rays, all target tokens are identical so the output collapses to the scene's mean color (Fig. 9, right). The scores slightly higher in PSNR than shuffled because a blurry mean outperforms a sharp but wrong view under MSE. With no ray, more sources no longer help in either PSNR or SSIM.
2.3 Bonus: Analysis and Exploration — Training View Count
Question
The baseline is trained with a fixed S = 2 source views but evaluated with S = 1, 2 and 4. How does training with a different number of source views, or a random number per step, change the evaluation with 1 to 6 source views?
Hypothesis is that a fixed-S model degrades away from its training S because of the sequence-length mismatch, and that training with a random S fixes this.
Experimental settings
I trained five models with the Q2.2 settings (100 epochs, batch size 4, same optimizer, schedule, seed, model and data) and changed only the training source view count: fixed S = 1, 2 (Q2.2 baseline), 4 and 6, and random S ~ U{1, 2, 4} drawn per step. Since each clip has 8 views, a larger S also means fewer targets (T = 8 − S). Each best-validation-PSNR checkpoint is evaluated on the validation split with S = 1 to 6 source views and correct target rays.
Results
The training view count matters by up to about 3.0 dB in PSNR and 0.02 in SSIM at the same evaluation S (Figure 11). Every multi-source model improves from evaluation S = 1 to S = 5, and the model trained with S = 4 is best or tied-best at every evaluation S. The S = 1 model gets worse with every added source, and the random-S model performs the same as the fixed S = 2 baseline. All models, including the one trained with S = 6, drop at evaluation S = 6. Figure 12 shows the gap visually: the S = 4 model draws more detailed and accurate predictions than the S = 1 model.




Conclusions
Training view count matters, but not because of a sequence-length mismatch, since the S = 4 model wins even at the lengths it never saw and random S gives nothing over the baseline. Performance improves as the training S increases up to S = 4 and then drops at S = 6. For this 8-view setup the best setting is to train with S = 4 and evaluate with as many sources as are available.
Possible explanations and limitations
More sources help because they cover more of the target and give the model parallax, but with 8 views per clip, S = 6 leaves only 2 targets per sample, so supervision drops to a third of S = 2. Random S adds nothing because sources are an unordered set in this architecture (shared positional embedding, no view-index embedding), so what is learned at one S transfers to any other. The S = 1 model never learned to choose between sources, so extra views only confuse it. The main limitations are a single seed per setting, short 100-epoch runs, and an evaluation target set that changes with S, which makes the drop at S = 6 partly an evaluation set effect.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
I overfit one fixed training clip with the default Mini-VGGT and shows the loss curve (total, camera and depth loss) in Fig. 13 and the screenshots comparing GT and prediction in Fig. 14.



3.2 Training and Evaluation
Training settings. I used the settings from configs/vggt.json (shown below) without changes.
| Setting | Value |
|---|---|
| Epochs | 1000 |
| Steps per epoch | 14 |
| Batch size | 16 |
| Loss | 5 · camera + 1 · depth |
| Optimizer | AdamW |
| Learning rate | 0.0006 |
| Weight decay | 0.01 |
| Betas | (0.9 0.95) |
| Warmup steps | 10 |
| Gradient clipping | 1.0 |

| Split | AbsRel ↓ | δ1 ↑ | Rot AUC@5 | Rot AUC@15 | Rot AUC@30 | Trans AUC@5 | Trans AUC@15 | Trans AUC@30 | Joint AUC@5 | Joint AUC@15 | Joint AUC@30 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Val | 0.152 | 0.797 | 0.238 | 0.574 | 0.742 | 0.017 | 0.081 | 0.186 | 0.007 | 0.066 | 0.168 |
| Test | 0.169 | 0.763 | 0.231 | 0.569 | 0.748 | 0.008 | 0.069 | 0.192 | 0.003 | 0.051 | 0.174 |
best.pt.

09c1414f1b, ground truth (left) and prediction (right).

21d970d8de, ground truth (left) and prediction (right).

280b83fcf3, ground truth (left) and prediction (right).

2b1dc6d6a5, ground truth (left) and prediction (right).Errors on held-out scenes. Depth transfers well to new rooms with AbsRel 0.15 for validation and 0.17 for test, and δ1 is around 0.8 for both. Its errors are the smooth inflated surfaces, discontinuities and missing thin structures seen in Fig. 16-19, which is expected from a 14px patch-level prediction. Rotation performs reasonably well (8.7° mean pairwise error), but translation direction is poor (40° mean pairwise error) and pulls the joint pose AUC@30 down to 0.17. The camera baselines in clips are short so a small absolute translation error becomes a large angular error. The training camera loss is still falling at epoch 1000 while validation has flattened, so more epochs or a larger translation weight would help improve the pose metrics.
3.3 Bonus: Analysis and Exploration — Point-Map Prediction
Question
I explore point-map prediction. The baseline gets points by unprojecting predicted depth with predicted cameras, so camera errors move whole depth maps. My question is: does the direct point-map prediction head give better point predictions than the camera + depth?
Experimental settings
I added a PointHead that copies the depth head with 3 output channels (dense linear prediction, RGB-guided refinement, patch-level confidence) and a point_loss that mirrors the depth loss with the per-pixel Euclidean distance to the GT points as the error. The total loss is 5 camera + 1 depth + 1 points. Everything else is identical to Q3.2. For every GT-valid pixel I compare two point reconstructions: predicted cameras + predicted depth (the baseline route) and the direct point head.
Results
The direct point head does not beat the camera + depth of the same model: 0.231 vs 0.225 on val and 0.218 vs 0.212 on test, about 2–3 % worse, with the same ranking in every view (Figure 20, right). Even in view 0, where there is no camera error to bypass, the head is slightly worse (0.125 vs 0.116). Adding the point loss does help the other tasks. The joint model improves depth AbsRel from 0.152 to 0.138 in validation and 0.169 to 0.151 in test, its own camera + depth points by 0.044 in validation and 0.024 in test, and joint pose AUC@30 from 0.17 to 0.18 in validation and 0.20 in test. Figures 21-22 show the two reconstructions are nearly identical in layout, with the point-head cloud slightly more scattered.
| Model | Split | Point error: cam + depth | Point head | AbsRel ↓ | δ1 ↑ | Rot err ↓ | Rot AUC@30 | Trans AUC@30 | Joint AUC@30 |
|---|---|---|---|---|---|---|---|---|---|
| camera + depth (Q3.2) | val | 0.269 (0.54 m) | — | 0.152 | 0.797 | 11.0° | 0.742 | 0.186 | 0.168 |
| + point head | val | 0.225 (0.45 m) | 0.231 (0.47 m) | 0.138 | 0.842 | 9.4° | 0.748 | 0.196 | 0.181 |
| camera + depth (Q3.2) | test | 0.236 (0.47 m) | — | 0.169 | 0.763 | 8.7° | 0.748 | 0.192 | 0.174 |
| + point head | test | 0.212 (0.41 m) | 0.218 (0.42 m) | 0.151 | 0.832 | 9.2° | 0.742 | 0.216 | 0.201 |




09c1414f1b with the joint model: GT (left), points from predicted cameras + depth (middle) and the direct point head (right).


2b1dc6d6a5: GT (left), points from predicted cameras + depth (middle) and the direct point head (right).Conclusions
A direct point head is not a better way to get points in this setup. The best points come from the joint model's cameras + depth while the direct model is close second. The point loss is nevertheless worth adding as an auxiliary task, since it improves depth, points and pose at once for more time per epoch.
Possible explanations and limitations
Both heads read the same tokens, so the point head only knows the pose that is already in them and can't beat the dedicated camera head. Regressing absolute coordinates per patch also adds noise. The point loss helps the other tasks because it is the only term that couples pose and depth, pushing the shared features towards a pose and depth that agree. The limitations are one seed per model, the simple point head, and the fact that camera error dominates the point error for both methods, so a better camera head would matter more than any change to the point head.