Andrew ID: robertre
Unprojected points and cameras of the 8 sampled views in the original world frame (meters). Scene 036bce3393_frame_004470.
The same views after canonicalization. First camera center: (0.0000, -0.0000, 0.0000). Valid-point centroid distance: 1.000 normalized units (scene scale 1.867 m).
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011279189493507147, "max_ray_reprojection_error_px": 0.00010904046939685941, "centroid_distance": 1.0},
{"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552},
{"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0},
{"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00010924567322945222, "max_ray_reprojection_error_px": 0.00010257320536766201, "centroid_distance": 1.0}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Canonicalization: Canonicalization changes the reference frame so that every camera pose and point is expressed in the frame of the first sampled camera of the scene. With this operation, the first camera has identity rotation and zero translation. By canonicalizing the cameras, we also normalize the scale by dividing translation, depth, and points by the distance from camera 0 to the centroid of all valid points. Canonicalization is necessary to remove the arbitrary world frame and scale of each scene so the training use samples with consistent conventions.
World-space ray directions: To obtain world space rays, we first back-project a given pixel to camera space, and rotate it to world space using the camera to world transformation. We then normalize it by scaling it to unit length. The advantage of using world space rays is that all the rays from different cameras share the same reference frame which is the first sampled camera's canonical frame.
Plücker moment: A ray has a camera center and a unit direction (o, d). The plucker coordinates describe the ray with 6 numbers (d, o × d). The cross product is used instead of just the camera center because the cross product does not depend on which point we pick as an origin. Mathematically, (o + t*d) × d = o × d. Therefore, using just (o, d) could have ambiguity, while (d, o × d) is unique for each ray.
Overfit on one fixed batch for 2000 steps. Final PSNR: 30.93 dB (SSIM 0.870).
Rows: target views. Columns: source 1, source 2, ground truth, prediction.
| Model | patch size 8, width 384, 8 transformer blocks, head dim 64, MLP ratio 4, QK-norm (14.6M parameters) |
| Views | 8 per sample (2 source, 6 target), from a 32-frame window, 224 x 224 |
| Batch size | 4 samples (24 target images) |
| Loss | MSE |
| Optimizer | AdamW, learning rate 2e-4, weight decay 0.05, betas (0.9, 0.95), gradient clipping 1.0, bf16 autocast |
| Schedule | 200 warmup steps, then cosine decay to 10% |
| Training length | 100 epochs (5,600 steps), 53 min on one A100 40 GB |
| Checkpoint | best validation PSNR (epoch 99) |
| Evaluation | validation split, 8 views from the first 32 prepared frames, seed 825 |
| Source views | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | 15.19 / 0.558 | 11.80 / 0.487 | 12.48 / 0.524 |
| 2 | 16.12 / 0.560 | 11.49 / 0.472 | 12.44 / 0.525 |
| 4 | 16.87 / 0.575 | 11.58 / 0.472 | 12.46 / 0.536 |
Validation batch during training. Left: epoch 0. Right: epoch 95. Columns: source views, ground truth, prediction.
| Correct rays | Shuffled rays | No rays | |
|---|---|---|---|
| S=1 | ![]() | ![]() | ![]() |
| S=2 | ![]() | ![]() | ![]() |
| S=4 | ![]() | ![]() | ![]() |
Columns in each image: source views, ground truth, prediction.
Number of source views: Having more input views improve the novel view synthesis. However, the gain will get smaller as more views are added. Since attention can accept any number of source view tokens, the model can take in more than 2 views even though it was only trained on 2 views. More source views give the model more information to infer the global geometry and appearance, and therefore help the model generalize better to novel views.
Ray-conditioning ablation: The model is clearly using the target ray information. Based on the ablation, the model performs a lot better when the correct target view ray information is given. At 2 context views, correct rays give 16.12 dB, no rays 12.44 and shuffled rays 11.49. Shuffled rays confuse the model even more and perform less than no rays.
Question: How does adding an SSIM term to the MSE loss change PSNR, SSIM and visual sharpness?
Settings: Same as 2.2, but the training loss is MSE + 0.2 · (1 - SSIM). The validation loss is still MSE.
| Source views | MSE (2.2) | MSE + SSIM | ||
|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | |
| 1 | 15.19 | 0.558 | 14.74 | 0.575 |
| 2 | 16.12 | 0.560 | 15.37 | 0.579 |
| 4 | 16.87 | 0.575 | 15.74 | 0.592 |
Correct target rays, validation split.
S=2. Left: MSE. Right: MSE + SSIM.
Results and conclusions: Including the SSIM term in the loss gives a tradeoff. It improved the structural similarity of the image but lowered the PSNR because PSNR scores the per pixel error directly and was downweighted due to the SSIM term. Visually, the predicted images with the SSIM term are much smoother, and the blocky patterns from the model without the ssim loss mostly disappeared. This test used only one ssim weight of 0.2 so the tradeoff might be different with other weights
One fixed training clip (8 views), 40 epochs of 25 steps, learning rate 1e-3. Final: total loss -0.159 (camera 0.002, depth -0.169), depth AbsRel 0.038, δ1 0.954, pose AUC@30 0.998.
Top: ground truth (left) and predicted cameras and depth (right), seen from behind the cameras and from above. Bottom: RGB, ground-truth depth and predicted depth for all 8 views.
| Model | frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (16 blocks), width 384, 6 heads, camera head with 4 cross-view blocks, depth head with RGB refinement (57.7M trainable parameters) |
| Views | 8 per sample, from a 32-frame window, 224 x 224 |
| Batch size | 16 |
| Loss | 5 x camera loss + 1 x depth loss |
| Optimizer | AdamW, learning rate 6e-4, weight decay 0.01, gradient clipping 1.0, bf16 autocast |
| Schedule | 10 warmup epochs, then cosine decay to 5% |
| Training length | 1000 epochs (14,000 steps), 3.3 h on one A100 40 GB |
| Checkpoint | best validation loss (epoch 843) |
| Evaluation | 8 views from the first 32 prepared frames, 224 px, seed 825, 28 scenes per split (784 view pairs) |
| Split | AbsRel | δ1 | Rotation AUC | Translation AUC | Pose AUC | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5° | 15° | 30° | 5° | 15° | 30° | 5° | 15° | 30° | |||
| Validation | 0.132 | 0.849 | 0.196 | 0.538 | 0.715 | 0.018 | 0.078 | 0.191 | 0.005 | 0.056 | 0.171 |
| Test | 0.156 | 0.809 | 0.179 | 0.511 | 0.718 | 0.014 | 0.074 | 0.177 | 0.002 | 0.052 | 0.157 |
For each scene: ground truth (left) and predicted cameras and depth (right), seen from behind the cameras and from above, then RGB, ground-truth depth and predicted depth for all 8 views.
On held-out scenes the model recovers the overall room layout, but test is slightly worse than validation. Translation is much worse than rotation since the cameras are very close together, so small position errors cause large direction errors. The predicted depth is smooth and blocky so thin structures like chair legs merge into blobs.
Question: How does the number of input views affect accuracy, runtime and memory?
Settings: The 3.2 model (trained with 8 views), evaluated on the validation split with 2, 4, 8, 16 and 32 views from the first 32 prepared frames. Batch size 1 on one A100 40 GB. Time is the model forward pass per sample, after one warm-up pass. Memory is the peak allocated GPU memory, including the model weights.
| Views | AbsRel | δ1 | Rotation AUC@30 | Translation AUC@30 | Pose AUC@30 | Time (ms) | Memory (GB) |
|---|---|---|---|---|---|---|---|
| 2 | 0.108 | 0.875 | 0.676 | 0.213 | 0.183 | 36 | 0.69 |
| 4 | 0.129 | 0.857 | 0.683 | 0.202 | 0.174 | 37 | 0.71 |
| 8 | 0.132 | 0.849 | 0.715 | 0.191 | 0.172 | 39 | 0.75 |
| 16 | 0.154 | 0.814 | 0.677 | 0.170 | 0.145 | 43 | 0.83 |
| 32 | 0.160 | 0.793 | 0.681 | 0.164 | 0.145 | 50 | 0.97 |
Results and conclusions: As input views increase, the runtime and memory increase as expected because the attention needs to handle more tokens. More views don't always mean better accuracies and the rotation AUC@30 starts to drop after 8 views. The translation and pose AUC drops as view counts increase as well. AbsRel rises from 0.108 with 2 views to 0.160 with 32 views. Therefore, more views could introduce challenge for the model for accurate geometry prediction. The limitation of this evaluation is that the model was only trained on 8 views, and that the evaluation for each view count use different frames. Therefore, this comparison is not perfectly apples-to-apples.