ScanNet++ world coordinates (meters) on the left, and canonical coordinates on the right. Each canonicalization side panel shows the first camera center, always (0, 0, 0), and the valid-point centroid distance, always 1.000.
| Unprojection | Canonicalization |
|---|---|
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
check_data.py output{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 9.187007526634261e-05, "max_ray_reprojection_error_px": 8.217109279939905e-05, "centroid_distance": 1.0},
{"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 9.650505671743304e-05, "max_ray_reprojection_error_px": 9.281559323426336e-05, "centroid_distance": 1.0},
{"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 0.0001078959321603179, "max_ray_reprojection_error_px": 9.650505671743304e-05, "centroid_distance": 1.0},
{"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 8.110157796181738e-05, "max_ray_reprojection_error_px": 9.013435192173347e-05, "centroid_distance": 1.0}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
In order to canonicalize, we start by putting each sample in the first camera's frame. Then, we multiply every camera-to-world pose by the inverse of first camera's pose, Tref_from_i = (Tworld_from_0)-1 Tworld_from_i. Note that cam 0 becomes the identity matrix as a result. Note, the depths are divided by a shared scale of the dist from cam 0 to the centroid of all valid points. This is also true for translations and points. For the world-space ray directions, for every pixel (u, v), we normalize such that d = normalize(R K-1[u, v, 1]T), such that K-1[u, v, 1]T is the direction through pixel center. This is all within the camera coords. The camera-to-world rotation, R, then rotates it into the canon frame. For the Plucker moment, we can simply state moment is the ray origin crossed (cross product) with the unit direction. By construction, this is the same for all points on the ray.
The final overfit PSNR is 33.03 dB. notice that there are some visible patch-grid artifacts.
We used the same vals given in configs/lvsm.json. we did make some changes though... we made batch size 3 rather than 4 and turned mixed precision off. did this since batch size of 4 ran out of memory on my 12 GB TITAN Xps..., and without native bf16 support mixed precision used more memory rather than less. Training on one gpu took 5.61 hours...
Validation PSNR and SSIM.
Correct rays
Shuffled rays
No rays
Epoch 90. Columns are source views, ground truth and prediction.
Correct rays. Columns are source views, ground truth and prediction.
1 source view.
2 source views.
4 source views.
Unsurprisngly, having more source views imrpoves the PSNR. this is with training usign only 2.
We start with correct rays.... they beat both ablations by 2.7 to 5.3 dB. we also record that gap increases with more source views, which is expected. Increasing the count of source views also doesn't help if we do not have the correct rays. Furthermore wihtout anyrays, everytarget gets the same input. As a result, our model outputs one flat average image for all targets. Thus, with shuffled rays, we produce a "better" image, even though the viewpoint is wrong. we note that this scores poorly. By construction, given an MSE loss, misplaced structure costs more than a flat average.
Effect of patch size on quality, runtime and memory.
A second model with patch size 16 instead of 8, all else equal (one run each).
From our results, patch size of 16 has slightly lower validation results (PSNR 16.01 dB SSIM 0.5559). Note that this is against 16.17 dB (SSIM 0.5590) patch size of 8... We note that it trains in 0.82 hours instead of 5.61. another by product is, we end up using 2.73 instead of 9.52 GB of peak memory. With 1, 2 and 4 source views, it reached 15.11, 16.01 and 16.65 dB. We still trend the same as as patch 8.
Second and third target views of the 2-source image in 2.2, with PSNR against ground truth.
Patch 8 gains only 0.16 dB... for ~7 times the training time and 3.5 times the memory. It shows less of a patch grid, but neither model recovers recognizable objects.
At this training length, patch 16 is probably the better trade-off, though both models were still improving, and each setting has one run, so the 0.16 dB gap may not be worthwhile.
Overfit loss curves for total, camera, and then depth.
| Ground truth | Prediction |
|---|---|
![]() | ![]() |
We left configs/vggt.json as is and trained on 4 GPUs, which took 8.1 hours, with best.pt coming from epoch 858.
Training loss curves for total, camera, and then depth.
Depth AbsRel is 0.141 on validation and 0.153 on test, and δ1 is 0.840 and 0.812. AUCs at 5° / 15° / 30°, validation then test, are 0.203 / 0.529 / 0.700 and 0.164 / 0.488 / 0.692 for rotation, 0.012 / 0.073 / 0.178 and 0.009 / 0.056 / 0.178 for translation, and 0.003 / 0.056 / 0.155 and 0.001 / 0.042 / 0.164 for joint pose.
| Ground truth | Prediction |
|---|---|
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
Depth shows a δ1 of 0.84 on val and 0.81 for test. we qualatitively notie that large planer structures are correctly reconstructed. Unsurprisngly, small/thin objects are obliterated. the maps themselves are quite box-like for the patch scale. On scale, it is sometimes off, as we see some scenes are reconstructed smaller than intended... On that note, rotations are good, with AUC@30 of 0.7. However, translation is lacking, with 0.18 AUC@#30, most likely because many view pairs have short baselines.
Effect of the number of test time views on accuracy, runtime and memory, for a model trained with 8 viesw.
We eval our trained model on the full val and test, expcet this time, with 2, 4, 8, 16 and 32 views. We use the same settings to ensure a 1-1 comparison. since the view sets are nested, we have each run with 16 or more views also scored over the first 16 frames.
Depth AbsRel and joint pose AUC@30, then forward time and peak memory.
16 and 32 views.
Validation
Test
2, 4, 8, 16 and 32 views.
Validation
Test
Cost
Firstly, with more views the val AbsRel rises from 0.115 to 0.173 for depth. Similarly, test shows AbsRel from 0.124 to 0.166... even on the same 16 frames, it still rises from 0.167 to 0.174 and from 0.163 to 0.166 between 16 and 32 views. Interestingly enough pose barely changes. Rotation AUC@30 stays at 0.68 to 0.71 and translation AUC@30 at 0.16 to 0.19. Some additional metrics that we have recorded would be time per view, which stays at ~20ms. but memory above the weights grows from 0.02 GB at 2 views, to a much larger 1.72 GB at 32... we suspect it is due to quadratic explosion due to attention.
Adding views during test makes the depth output worse and doesn't even improve pose. Furthermore, memory grows roughly quadratically. This suggests the model does not generalize well beyond its training view count, though since this is one checkpoint, without any retraining, it does not show the effect of training with a different count. Also, the 2-view pose results use only 28 pairs, while the same-frame check only covers 16 and 32 views.
As a thought experiment, we take the same 16 frames and add more views. Since the depth targets are normalized by the centroid of all the views' points, the scale of said 16 frames shifts with the number of views. As a result, AbsRel punishes this shift. Going past the 8 training views also puts the model outside its training distribution, which likely adds to it. In a similar vein, for pose, extra views don't change the relative geometry of the pairs we score, thus pose stays flat.