Contents
| After unprojection (world frame, meters, z up) | After canonicalization (camera 0 frame, normalized units) |
|---|---|
![]() | ![]() |
Scene 079a326597
Both steps put the scene where it should be. Unprojection merges the 8 views in the ScanNet++ world frame, and canonicalization moves them to camera 0's frame at unit scale. The first camera goes from (3.76, 2.08, 1.53) m to (0, 0, 0), and the valid-point centroid distance becomes 1 (scene scale 2.080 m). So the scene is in the frame the models expect.
Interactive viewer (open in new tab).
split train | dataset_samples 224 | checked_samples 4 e050c15a8d_frame_007080 | valid_depth 0.978 | max_reproj_px 1.10e-04 | max_ray_reproj_px 9.41e-05 | centroid 1.0000 5f99900f09_frame_002820 | valid_depth 0.990 | max_reproj_px 1.11e-04 | max_ray_reproj_px 1.11e-04 | centroid 1.0000 47b37eb6f9_frame_002720 | valid_depth 0.961 | max_reproj_px 9.35e-05 | max_ray_reproj_px 8.90e-05 | centroid 1.0000 4c5c60fa76_frame_006690 | valid_depth 0.752 | max_reproj_px 1.07e-04 | max_ray_reproj_px 9.42e-05 | centroid 1.0000 batch shapes: rgb [2, 8, 3, 224, 224], depth [2, 8, 1, 224, 224], depth_valid [2, 8, 1, 224, 224], K [2, 8, 3, 3], T_world_from_camera [2, 8, 4, 4], points_world [2, 8, 3, 224, 224], ray_origins [2, 8, 3, 224, 224], ray_directions [2, 8, 3, 224, 224], plucker [2, 8, 6, 224, 224], frame_numbers [2, 8], original_world_from_reference [2, 4, 4], scene_scale [2]
Canonicalization. All poses are expressed relative to camera 0, T_i' = T_0^-1 T_i, so camera 0 is the identity. Translations, depths and points are then divided by the distance from camera 0 to the valid-point centroid, so that distance is 1. This removes the arbitrary world frame and scale of each scene.
World-space ray directions. For pixel (u, v), d = normalize(R K^-1 [u, v, 1]), with R the camera-to-world rotation. No translation is applied since d is a direction; the ray origin is the camera center.
Plücker moment. m = o x d. It is the same for any point on the ray, since (o + s d) x d = o x d, so (d, m) encodes the line itself. With unit d, |m| is the line's distance from the origin.
After 2000 steps the final PSNR is 33.97 dB (SSIM 0.914).
| val_epoch000.png (columns: source 1, source 2, ground truth, prediction) |
|---|
![]() |
| Training curves |
|---|
![]() |
Training improves steadily but does not converge in 100 epochs. The curve flattens near the end probably because the cosine learning rate decays. Val PSNR rises from 12.3 to 16.1 dB, and the same config trained for 300 epochs (about 3 hours) gets 15.68 / 16.92 / 17.92 dB at S = 1 / 2 / 4. So the model is underfit; the rest of this section uses the 100-epoch model.
Settings. I used the base configs/lvsm.json but decreased batch size to 2 so I could train on my e-gpu. That gives 112 steps per epoch and 11,200 steps in total, and it took about 1 hour.
| Validation | correct rays | shuffled rays | no rays |
|---|---|---|---|
| S = 1 | 15.17 / 0.557 | 11.76 / 0.486 | 12.46 / 0.527 |
| S = 2 | 16.09 / 0.560 | 11.45 / 0.473 | 12.44 / 0.527 |
| S = 4 | 16.69 / 0.575 | 11.60 / 0.477 | 12.48 / 0.539 |
PSNR (dB) / SSIM on the validation split, best.pt (epoch 95).
| Training visualizations (validation batch during training): epoch 0, 50, 90 |
|---|
![]() |
| S = 1, correct | S = 2, correct | S = 4, correct |
|---|---|---|
![]() | ![]() | ![]() |
| S = 2, shuffled | S = 2, no rays | |
![]() | ![]() |
Evaluation visualizations. Columns: source views, ground truth, prediction.
Number of source views. More source views improve PSNR. More views likely show more of each target, so the model guesses less. PSNR is 15.2 dB at S = 1, 16.1 at S = 2 and 16.7 at S = 4, even though I trained only with S = 2. So the model benefits from view counts it never saw in training.
Ray conditioning. The target rays tell the model which view to render. Without them it can't know the viewpoint and outputs a blurry average of the scene colors. Correct rays give 2.7 to 4.2 dB more than no rays (about 12.5 dB), and shuffled rays are worse still (about 11.5 dB), likely because under MSE a structured guess at the wrong viewpoint costs more than a flat average. So pose conditioning matters more than the number of source views.
The predictions are blocky even with correct rays. The 8x8 patches, the small model and the MSE loss, which averages over uncertain details, are the likely causes. You can see the layout of the room if you squint but there is no fine texture, and targets that see little of the sources are blurrier. 2.3 tests the loss part.
Question. Does adding a perceptual loss to the MSE make the blocky 2.2 outputs sharper, and what does it cost in PSNR and SSIM?
Settings. I follow the same setup as 2.2, with the loss MSE + w * perceptual (configs/lvsm_perceptual.json, configs/lvsm_perceptual_w01.json). I used LVSM's VGG-19 feature loss. I tried w = 0.5, the paper's value for scenes, and w = 0.1, which makes the two terms about equal.
| Training curves (loss is the MSE term only) |
|---|
![]() |
| Validation, correct rays | S = 1 | S = 2 | S = 4 | Sharpness (S = 2) | Border ratio (S = 2) | LPIPS Alex / VGG (S = 2) |
|---|---|---|---|---|---|---|
| MSE (2.2) | 15.17 / 0.557 | 16.09 / 0.560 | 16.69 / 0.575 | 0.29 | 10.8 | 0.677 / 0.614 |
| MSE + 0.1 perceptual | 15.17 / 0.562 | 16.06 / 0.562 | 16.68 / 0.571 | 0.54 | 1.83 | 0.661 / 0.628 |
| MSE + 0.5 perceptual | 14.93 / 0.559 | 15.65 / 0.559 | 16.05 / 0.568 | 0.50 | 1.61 | 0.687 / 0.648 |
PSNR (dB) / SSIM on the validation split. Sharpness is the mean image gradient relative to GT (GT = 1). Border ratio is the gradient across the 8 px patch borders over the gradient inside patches (GT = 1.01), so a large value means a visible grid. LPIPS: lower is better.
| Same validation targets, S = 2, correct rays |
|---|
![]() |
Results and conclusions. At w = 0.1 the perceptual term removes most of the patch grid without losing PSNR or SSIM. VGG features likely penalize grid patterns that MSE barely notices. At S = 2 the border ratio falls from 10.8 to 1.83 and sharpness rises from 0.29 to 0.54, while PSNR stays within 0.03 dB and SSIM within 0.006 at every S. So w = 0.1 is a clear improvement in blockiness.
A larger weight does not seem to help. Too much perceptual weight likely takes away from pixel accuracy. w = 0.5 loses 0.44 dB at S = 2 and 0.63 dB at S = 4 and is no sharper (0.50). So the paper's weight is worse than w = 0.1 for this small model.
The term reduces blockiness but not blur. The blur is likely due to the small, underfit model and to details the sources do not determine. No new texture appears, sharpness stays near half of GT, and LPIPS shows no clear gain (AlexNet 0.677 to 0.661, VGG 0.614 to 0.628). So the loss alone does not fix the 2.2 outputs.
Limitations. One run per weight and only two weights. Sharpness measures edge strength, not correctness. A faint grid remains (border ratio above 1.01).
| Overfit loss curve |
|---|
![]() |
| Ground truth | Prediction |
|---|---|
![]() | ![]() |
The model fits a single clip almost exactly. With one clip there is nothing to generalize to, so errors should end near zero. After 40 epochs of 25 steps (default overfit settings) the total loss drops from 5.5 to below zero, AbsRel is 0.036, rotation error is 0.24 degrees and pose AUC@30 is 0.99, and the predicted cameras sit on the GT ones. So it can learn both cameras and depth.
Interactive viewer (open in new tab). Switch between ground truth and prediction in the second menu.
Settings. I used configs/vggt.json but decreased the batch size to 4 because of the same memory limit on my e-gpu as earlier. I kept 8 views and sampling range 32. I kept the learning rate and the per-epoch schedule, so training ran 56 steps per epoch (56,000 steps in total) and took roughly 6 hours.
| Training loss_curve.png (total, camera, depth) |
|---|
![]() |
Depth is learned early and camera pose late. Depth gets most of its gain right away, while the cameras keep improving for hundreds of epochs. The val depth loss drops from 3.05 to 0.35 in 50 epochs and reaches 0.28 by epoch 1000, while the val camera loss goes from 0.18 at epoch 50 to 0.13 at 500 and flattens near 0.12 from about epoch 700. So the long training likely helps the cameras most.
| Split | AbsRel | delta1 | Rot AUC 5 / 15 / 30 | Trans AUC 5 / 15 / 30 | Pose AUC 5 / 15 / 30 | Rot / trans error (deg) |
|---|---|---|---|---|---|---|
| val | 0.141 | 0.858 | 0.169 / 0.486 / 0.674 | 0.019 / 0.075 / 0.171 | 0.006 / 0.062 / 0.152 | 13.1 / 43.1 |
| test | 0.158 | 0.810 | 0.161 / 0.456 / 0.678 | 0.016 / 0.076 / 0.174 | 0.003 / 0.044 / 0.150 | 10.8 / 40.1 |
28 val and 28 test scenes, 8 views each, best.pt. The test set was evaluated once.
| Ground truth | Prediction |
|---|---|
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
![]() | ![]() |
The first four validation scenes (09c1414f1b, 21d970d8de, 280b83fcf3, 2b1dc6d6a5).
Interactive viewer (open in a new tab). Switch between ground truth and prediction in the second menu.
Errors on held-out scenes. Depth generalizes fairly well. The depth maps are smooth, so coarse structure survives but small objects lose their shape. AbsRel is 0.14 on val and 0.16 on test (delta1 0.86 and 0.81), and all four scenes have the right room layout, but the table and chairs in the first scene merge into one blob. So depth errors are mostly in fine detail.
Cameras are the weak part, mainly translation. Pose AUC takes the worse of rotation and translation, so translation sets it. Rotation AUC@30 is about 0.68 (mean error 11 to 13 degrees), translation about 0.17 (40 to 43 degrees) and pose 0.15, and in the first scene the predicted cameras collapse into one cluster while GT camera 1 sits apart. This is consistent with the model underestimating the baselines.
Question. Does a direct point-map head avoid the camera error found in 3.2, and does training it change the camera and depth results?
Settings. Same as 3.2, plus a point head built like the depth head, with a VGGT-style confidence-weighted point loss plus a gradient term at weight 1. Training took roughly 7 hours. All results are on val.
| Mean point error (m) | 3.2 model | With point head |
|---|---|---|
| Predicted cameras + predicted depth | 0.550 | 0.500 |
| Direct point head | 0.493 | |
| Predicted cameras + GT depth | 0.451 | 0.399 |
| GT cameras + predicted depth | 0.307 | 0.276 |
| AbsRel | delta1 | Rot AUC@30 | Trans AUC@30 | Pose AUC@30 | Rot / trans error (deg) | |
|---|---|---|---|---|---|---|
| 3.2 model | 0.141 | 0.858 | 0.674 | 0.171 | 0.152 | 13.1 / 43.1 |
| With point head | 0.127 | 0.862 | 0.747 | 0.191 | 0.179 | 10.6 / 41.1 |
| Validation curves |
|---|
![]() |
| Top-down view of the point clouds (x, z in the camera 0 frame, same axes per row, point-head model) |
|---|
![]() |
Results and conclusions. The direct head does not avoid the camera error. Points in camera 0's frame likely still need the relative pose, so the head faces the same problem as the cameras. It gets 0.493 m against 0.500 m for the same model's unprojected points, while GT cameras give 0.276 m. So predicting points directly is not a way around the weak cameras.
Training the head seems to help the other tasks. The point loss likely adds geometry supervision to the shared layers. AbsRel improves from 0.141 to 0.127 and rotation AUC@30 from 0.674 to 0.747. Averaged over epochs 950 to 1000, to rule out the later checkpoint (epoch 997 against 833), AbsRel is 0.130 against 0.151, rotation AUC@30 is 0.742 against 0.675 and translation AUC@30 is 0.186 against 0.175. So the head is more useful as an extra loss than as a point output.
Limitations. One run per model and one point weight, so part of the gain could be seed noise. The direct points show horizontal banding, mostly in far regions for some reason.
Second exploration: number of views
Question. How do accuracy, runtime and memory change with 2, 4, 8 or 16 views at evaluation time, using the 8-view model?
Settings. The 3.2 model on the full val split with the 3.2 settings, changing only V. Runtime is the forward pass per scene.
| V | AbsRel | delta1 | Rot AUC@30 | Trans AUC@30 | Pose AUC@30 | Pose pairs | ms / scene | Peak GB |
|---|---|---|---|---|---|---|---|---|
| 2 | 0.123 | 0.843 | 0.704 | 0.152 | 0.136 | 28 | 13 | 0.76 |
| 4 | 0.134 | 0.856 | 0.685 | 0.166 | 0.146 | 168 | 20 | 0.85 |
| 8 | 0.141 | 0.858 | 0.674 | 0.171 | 0.152 | 784 | 33 | 1.01 |
| 16 | 0.159 | 0.814 | 0.655 | 0.168 | 0.139 | 3360 | 62 | 1.35 |
| Accuracy and cost against the number of views |
|---|
![]() |
Results and conclusions. Using more views than in training does not seem to help. All views come from the same 32 frames, so 16 views likely adds near-duplicates more than new information. 16 views is worse than 8 on every metric (AbsRel 0.159 against 0.141, delta1 0.814 against 0.858). So the model gains nothing past its training view count.
Cost grows about linearly with V. At this size the per-frame parts (DINO, frame attention, heads) likely dominate over the quadratic global attention. Runtime goes from 13 to 62 ms per scene and memory from 0.76 to 1.35 GB. So cost is not what limits the view count here.
Limitations. One checkpoint and one seed. Camera metrics average over 1 pair per scene at V = 2 and 120 at V = 16, so the V = 2 values are noisier.