Scene 1: 
uv run python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000 --output outputs/unprojectionScene 1: 
uv run python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization
for V sampled views of the same scene, the first camera
view, V[0] is used as the reference for the world, and all
other camera frames in V are predicted relative to this
frame.
By default, the center of each camera serves as the origin for all its rays and each pixel has its corresponding direction. In World-space, the origin of the first camera view serves as the origin for all other camers, using the relative transforms calculated for canonicalization.
3D lines are usually represented by an origin o and
direction d. But the same 3D line can be represented with a
different origin o1 and the same direction d.
This introduces a dependency on the choice of origin. To decouple our
model training from this choice, we use Plücker moment, which is
(o x d) perpendicular to both o and
d. Any arbitrary choice of o will give the
same Plücker moment.
uv run python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010746094631031156,
"max_ray_reprojection_error_px": 9.423537994734943e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 9.281559323426336e-05,
"max_ray_reprojection_error_px": 7.968755380716175e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 9.951626998372376e-05,
"max_ray_reprojection_error_px": 8.800713840173557e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
uv run python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit
Mini-LVSM: 14.6M params, device cuda, S=2, 2000 steps/epoch, 1 epochs
epoch 0 | train loss 0.00687 psnr 25.38 | val loss 0.00087 psnr 30.90 ssim 0.8698 | 1475s, targetsTraining Progress:
| epoch 0 | epoch 50 | epoch 95 |
|---|---|---|
![]() |
![]() |
![]() |
uv run python -m models.lvsm.train --config configs/lvsm.jsonEvaluation Progress:
| 1 | 2 | 4 | |
|---|---|---|---|
| Corrected | ![]() |
![]() |
![]() |
| Shuffled | ![]() |
![]() |
![]() |
| None | ![]() |
![]() |
![]() |
Validation metrics (epoch 100):
| Source views (S) | Conditioning | PSNR (dB) | SSIM | MSE loss | # images |
|---|---|---|---|---|---|
| 1 | Correct | 15.20 | 0.5580 | 0.03515 | 196 |
| 1 | Shuffled | 11.81 | 0.4875 | 0.07277 | 196 |
| 1 | None | 12.48 | 0.5298 | 0.05873 | 196 |
| 2 | Correct | 16.14 | 0.5599 | 0.02936 | 168 |
| 2 | Shuffled | 11.49 | 0.4719 | 0.07785 | 168 |
| 2 | None | 12.45 | 0.5284 | 0.05903 | 168 |
| 4 | Correct | 16.86 | 0.5750 | 0.02478 | 112 |
| 4 | Shuffled | 11.61 | 0.4734 | 0.07809 | 112 |
| 4 | None | 12.47 | 0.5379 | 0.05863 | 112 |
In all 3 source view ablations, the correct source view gave the best performance, highlighting that the right ray bundle and Plücker moments are essential for learning good representation. Surprisingly None rays performed better than shuffled rays in PSNR, even though the images look more like a room in shuffled which is supported by SSIM. I believe the reason for this is that in None, all the rays now just attend to each other equally because every one of them is the same and just even out the final output through out the image.
The other takeaway is the improved performance from increased number of views. The model is able create a more accurate 3D representation in its latent and informing the desired view synthesis.
Because of Time and GPU constraints, I am evaulating the smaller patch size model primarily for the 25th epoch. From the regular patch size of 8, we get:
epoch 25 | train loss 0.03866 psnr 14.77 | val loss 0.03552 psnr 14.95 ssim 0.5457 | 33seach epoch takes about 31 seconds GPU memory: 9904 mb PSNR (4 Views) : 15.71 SSIM (4 Views) : 0.5558 Loss (4 Views) : 0.030
Running a smaller patch size model was not possible on my GPU with 16GB VRAM, so I had to use grad check and recompute all the forward values during my backward pass.
each epoch takes about 221 seconds (with grad checkpointing) GPU memory: 9896 mb (with grad checkpointing) PSNR (4 Views) : 15.07 SSIM (4 Views) : 0.5796 Loss (4 Views) : 0.034
p = 4 |
p = 8 |
|---|---|
![]() |
![]() |
While p = 4 is not as blocky as p = 8, it
does seem like it takes a lot longer for it to train, as the loss for
patch size 8 reduced more than patch size 4.

| Ground Truth | Prediction |
|---|---|
![]() |
![]() |
uv run python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfitThis model was trained for 850 epochs (didn't have more time to train it). The rest of the seeting were default values with 8 views and sampling_range = 32

| Ground Truth | Prediction |
|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
uv run python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit --850
uv run python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/metrics.json --visualization outputs/vggt_train/visualization/index.html
uv run python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split test --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/test_metrics.jsonValidation/Test Metric
| Metric | 3 deg | 5 deg | 15 deg | 30 deg |
|---|---|---|---|---|
| Rotation AUC | 0.0689 | 0.1497 | 0.4052 | 0.5812 |
| Translation AUC | 0.0072 | 0.0148 | 0.0510 | 0.1351 |
| Joint Pose AUC | 0.0017 | 0.0074 | 0.0355 | 0.1086 |
| Metric | Value |
|---|---|
| Depth AbsRel | 0.1749 |
| Depth δ1 | 0.7611 |
| Total loss | 1.0304 |
| Camera loss | 0.1398 |
| Depth loss | 0.3317 |
The model is doing good in terms of rotation, that's why the points are probably in the right space to begin with/ But the translation and joint pose AUCs are not looking great. More than 82% of them have an error greater than 30 deg.
The depth prediction has done pretty well.