Below are visualizations of the ScanNet++ multi-view point clouds and camera frustums before and after canonicalization. After canonicalization, Camera 0 is aligned with the origin at the identity pose [0, 0, 0] and the valid-point centroid is normalized to a distance of approximately 1.0 unit from the origin.
check_data.py)
Running python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 verifies tensor shapes, valid depth masks, reprojection errors (< 0.01 px), and canonicalized centroid distances (~1.0):
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010844774078577757,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.281559323426336e-05,
"max_ray_reprojection_error_px": 8.446346328128129e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010504279634915292,
"max_ray_reprojection_error_px": 9.423537994734943e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
To verify the patchification, token assembly, transformer forward pass, and photometric MSE loss, Mini-LVSM was overfit on a single fixed batch for 2,000 steps.
Final Overfit PSNR: 32.62 dB
val_epoch000.png)
Training Settings: Trained for 100 epochs on the 224 ScanNet++ training scenes at 224×224 resolution with batch_size = 4, num_views = 8, num_source_views = 2 (6 target views per sample), and eval_num_frames = 32. The Mini-LVSM architecture uses patch_size = 16, d_model = 384, n_layers = 8, head_dim = 64, mlp_ratio = 4.0, and use_qk_norm = true. Optimization used AdamW (lr = 2e-4, weight_decay = 0.05, betas = (0.9, 0.95)), 200 warmup steps, gradient clipping at 1.0, and bfloat16 mixed precision (amp = true) with float32 MSE loss.
| Number of Source Views | Ray Conditioning | Validation MSE Loss ↓ | Validation PSNR (dB) ↑ | Validation SSIM ↑ |
|---|---|---|---|---|
1 Source View (S = 1) |
correct |
0.0358 | 15.01 | 0.5521 |
1 Source View (S = 1) |
shuffled |
0.0729 | 11.80 | 0.4861 |
1 Source View (S = 1) |
none |
0.0602 | 12.39 | 0.4903 |
2 Source Views (S = 2) |
correct |
0.0302 | 15.89 | 0.5536 |
2 Source Views (S = 2) |
shuffled |
0.0778 | 11.49 | 0.4721 |
2 Source Views (S = 2) |
none |
0.0602 | 12.39 | 0.4927 |
4 Source Views (S = 4) |
correct |
0.0259 | 16.52 | 0.5669 |
4 Source Views (S = 4) |
shuffled |
0.0778 | 11.60 | 0.4743 |
4 Source Views (S = 4) |
none |
0.0599 | 12.41 | 0.5054 |
val_epoch000.png)
val_epoch095.png)
correct
shuffled
none
correct
shuffled
none
correct
shuffled
none
Discussion: Effect of Source Views & Ray Conditioning:
Increasing the number of source views from 1 to 2 and 4 can improve both PSNR and SSIM under correct ray conditioning, as additional views provide broader scene coverage and more visual information about regions that may be occluded in individual views. With only 1 source view, disoccluded regions and depth ambiguities make it difficult for the model to reconstruct unseen structures accurately, potentially producing blurrier boundaries and missing details. Additional source views provide complementary visual information that helps the transformer establish better correspondence between source observations and the target view.
In the ray-conditioning ablation, shuffled conditioning supplies Plücker rays from a different target camera pose instead of the correct target rays. This creates a mismatch between the target geometry provided to the model and the ground-truth target image, potentially leading to geometrically inconsistent predictions and lower PSNR/SSIM. When conditioning is set to none, all target rays are replaced with zeros. Consequently, the target query tokens lose explicit 3D camera-ray information and rely primarily on learned projection biases, 2D positional embeddings, and source-view features. This makes accurate target-view reconstruction more difficult and may result in blurrier or less geometrically consistent predictions.
Question & Experimental Setup:
How does the number of source views used during training affect Mini-LVSM's novel-view synthesis performance when evaluated with 1, 2, and 4 source views? To investigate this, I trained three models with fixed source-view counts: S = 1, S = 2 (baseline), and S = 4. All models used the same architecture, image resolution, patch size, optimizer settings, and 100-epoch training schedule. The number of source views during training was the experimental variable. Each trained checkpoint was evaluated with 1, 2, and 4 source views using correct target-ray conditioning.
| Training Configuration | Eval: 1 Source View (PSNR / SSIM) |
Eval: 2 Source Views (PSNR / SSIM) |
Eval: 4 Source Views (PSNR / SSIM) |
|---|---|---|---|
Fixed S = 1 |
14.980 / 0.5534 | 14.903 / 0.5476 | 14.666 / 0.5562 |
Fixed S = 2 (Baseline) |
15.012 / 0.5521 | 15.894 / 0.5536 | 16.519 / 0.5669 |
Fixed S = 4 |
15.079 / 0.5535 | 16.143 / 0.5579 | 17.071 / 0.5746 |
The following visualizations compare predictions from the three training configurations under correct target-ray conditioning. Each image is generated using the corresponding model checkpoint.
Conclusions & Discussion:
Training with more source views substantially improves performance when the model is evaluated with multiple source views. The model trained with S = 4 achieves the best results at every evaluation setting, reaching 17.07 dB PSNR and 0.5746 SSIM with four evaluation source views. Compared with the baseline trained using S = 2, this represents improvements of approximately 0.55 dB PSNR and 0.0077 SSIM at four-view evaluation.
The improvement is especially pronounced when evaluating with 2 or 4 source views. For example, at four-view evaluation, PSNR increases from 14.67 dB for the model trained with one source view to 17.07 dB for the model trained with four source views. This suggests that exposure to multiple source views during training helps the transformer learn to exploit complementary visual information and reconstruct target views more effectively.
In contrast, performance with only one evaluation source view changes relatively little across training configurations: PSNR ranges from 14.98 to 15.08 dB, and SSIM remains close to 0.553. This indicates that additional multi-view training primarily benefits settings where multiple source observations are available, rather than substantially improving single-view reconstruction.
A limitation of this experiment is that the number of target views available for supervision and evaluation decreases as the number of source views increases. The evaluation reports 196, 168, and 112 target images for 1, 2, and 4 source views, respectively. Therefore, the comparisons are informative but are not based on exactly identical sets of target images. Furthermore, only fixed source-view counts were evaluated; randomly sampling the number of source views during training remains an interesting direction for future exploration.
Mini-VGGT (frozen DINOv2 ViT-B/14 encoder + 8 alternating frame/global attention pairs + camera and RGB-refined depth heads) was first overfit on a single 8-view training clip to validate the alternating attention mechanism and the camera/depth loss formulations.
Training Settings: Trained using the full ScanNet++ training split with
num_views = 8, sampling_range = 32,
eval_num_frames = 32, and image resolution 224×224.
The model uses a DINOv2 ViT-B/14 encoder with embedding dimension 384,
6 attention heads, and 6 transformer layers. Training used 100 epochs,
a learning rate of 0.0006, and weight decay of 0.01. Camera and depth
losses were weighted by 5.0 and 1.0, respectively.
Evaluated on the full validation and test splits using
--num-views 8 --eval-num-frames 32 --image-size 224 --seed 825.
Each split contains 28 samples and 784 pose pairs.
| Metric | Validation | Test |
|---|---|---|
| Total Loss | 1.2011 | 1.0634 |
| Camera Loss | 0.1719 | 0.1404 |
| Depth Loss | 0.3416 | 0.3612 |
| Depth AbsRel | 0.1737 | 0.1940 |
| Normalized Depth RMSE | 0.2209 | 0.2452 |
| Depth δ1 | 0.7582 | 0.7253 |
| Valid Depth Pixels | 10,347,720 | 10,517,881 |
| Metric | Validation | Test |
|---|---|---|
| Rotation Error (degrees) | 26.9319 | 21.4851 |
| Translation Error (degrees) | 57.2157 | 56.6677 |
| Rotation AUC @ 3° | 0.0833 | 0.0821 |
| Rotation AUC @ 5° | 0.1227 | 0.1329 |
| Rotation AUC @ 15° | 0.2789 | 0.3106 |
| Rotation AUC @ 30° | 0.4417 | 0.4990 |
| Translation AUC @ 3° | 0.0009 | 0.0017 |
| Translation AUC @ 5° | 0.0020 | 0.0043 |
| Translation AUC @ 15° | 0.0321 | 0.0271 |
| Translation AUC @ 30° | 0.0922 | 0.0870 |
| Joint Pose AUC @ 3° | 0.0000 | 0.0000 |
| Joint Pose AUC @ 5° | 0.0013 | 0.0015 |
| Joint Pose AUC @ 15° | 0.0252 | 0.0168 |
| Joint Pose AUC @ 30° | 0.0790 | 0.0704 |
Discussion of Errors on Held-Out Scenes: Mini-VGGT generally recovers the coarse spatial structure of indoor scenes, including room layouts, planar surfaces, and camera motion. However, errors remain on held-out sequences. Qualitatively, depth predictions are less reliable in textureless regions, such as white walls and uniform tabletops, and around specular or reflective surfaces; these regions can exhibit depth distortions and local planar warping. Depth predictions also tend to smooth sharp discontinuities at object boundaries, occasionally producing thin trails of erroneous points between foreground objects and background surfaces after unprojection. Camera pose estimation is particularly challenging when views have limited visual overlap or small translational baselines, especially during rapid rotational motion. In these cases, translation-direction estimates can become unreliable, contributing to the low Translation AUC and Joint Pose AUC at strict angular thresholds. This is consistent with the held-out results: depth AbsRel is 0.1737 on validation and 0.1940 on test, while Joint Pose AUC at 5° is only 0.0013 and 0.0015, respectively. These results suggest that, although the model captures coarse scene structure, accurate depth boundaries and camera pose recovery remain difficult in challenging viewing conditions.
Question & Experimental Setup:
How does varying the number of input views V affect 3D
reconstruction accuracy, inference runtime, and GPU memory consumption?
I evaluated the trained Mini-VGGT checkpoint across
V ∈ {2, 4, 8} views sampled from the 32-frame evaluation
window on the ScanNet++ validation split. I compared depth metrics,
camera pose AUCs, total evaluation runtime, and sampled peak GPU memory.
The 8-view configuration is the training baseline.
| Number of Views (V) | Depth AbsRel ↓ | Depth δ1 ↑ | Joint Pose AUC @ 5° / 15° / 30° ↑ | Runtime (s) | Peak Sampled GPU Memory (MB) |
|---|---|---|---|---|---|
| 2 Views | 0.1332 | 0.8143 | 0.0000 / 0.0357 / 0.0988 | 17.38 | 1,060 |
| 4 Views | 0.1679 | 0.7559 | 0.0000 / 0.0270 / 0.0835 | 19.09 | 1,948 |
| 8 Views (Baseline) | 0.1737 | 0.7582 | 0.0013 / 0.0252 / 0.0790 | 34.94 | 4,564 |
Runtime and Memory: Increasing the input from 2 to 8 views increased the measured evaluation runtime from 17.38 s to 34.94 s, approximately a 2.01x increase. Sampled peak GPU memory rose from 1,060 MB to 4,564 MB, approximately a 4.31x increase. The 4-view configuration required 19.09 s and 1,948 MB. Thus, using more views substantially increased computational and memory costs in this experiment.
Conclusions & Limitations: In this experiment, increasing the number of input views did not produce monotonic improvements in reconstruction accuracy. The 2-view configuration achieved the best depth metrics, with an AbsRel of 0.1332 and a δ1 accuracy of 0.8143, compared with 0.1737 and 0.7582 for the 8-view baseline. It also achieved the highest Joint Pose AUC at 15° and 30°. The 8-view configuration performed slightly better at the strict Joint Pose AUC threshold of 5°, although all three configurations had very low scores at this threshold. These results suggest that, for this validation split and checkpoint, additional views did not necessarily provide more useful information and may introduce redundant or less informative observations. The computational cost increased with view count: the 8-view configuration took approximately twice as long as the 2-view configuration and used more than four times the sampled peak GPU memory. Consequently, the 2-view setting offered the strongest overall depth accuracy and a favorable runtime-memory trade-off in this experiment. However, pose metrics should be interpreted with care because the number of evaluated pose pairs differed across configurations (28, 168, and 784 for 2, 4, and 8 views, respectively).