Name: Yu-Rou Tuan (andrew id: yurout)


First camera center (0, 0, 0), valid-point centroid distance 1.000.
First camera center (0, 0, 0), valid-point centroid distance 1.000.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011141905270051211,
"max_ray_reprojection_error_px": 0.00010682854917831719,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001220703125,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.777836385183036e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011162471491843462,
"max_ray_reprojection_error_px": 0.00010746094631031156,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization. Every clip is moved into the coordinate frame of its first camera, so that camera always sits at the origin looking down +z. Then everything is scaled so the average of the valid 3D points is at distance 1 from it. Different scans have different origins, orientations and sizes, and this puts them all on the same footing.
World-space ray directions. For each pixel, the inverse intrinsics give the direction of its ray in camera coordinates. Rotating that by the camera's rotation gives the direction in world space. Only the rotation matters here, since directions don't depend on where the camera is.
Plücker moment. A ray is described by its direction d and its moment o × d, where o is the camera center. The direction alone can't tell apart two parallel rays coming from different cameras, but the moment can, since it depends on where the line is. It also doesn't change if you pick a different point on the same line. For the first camera the moment is zero, because that camera is at the origin.
Final overfit PSNR: 30.90 dB.
Columns: source views, ground truth, prediction.
Training settings: 300 epochs, batch size 4, 2 source views during training, AdamW with learning rate 2e-4 and weight decay 0.05, 200 warmup steps then cosine decay, gradient clipping at 1.0, bf16 mixed precision. The model uses patch size 8, width 384 and 8 transformer layers. Training took about an hour on one RTX 5090. I first trained for the default 100 epochs, but it was still improving, so I trained longer.
Training curves.
Validation visualization during training (epoch 290). Columns: source views, ground truth, prediction.
Validation PSNR / SSIM:
| Source views | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | 15.76 / 0.583 | 11.66 / 0.483 | 12.46 / 0.532 |
| 2 | 16.97 / 0.591 | 11.26 / 0.463 | 12.43 / 0.528 |
| 4 | 18.09 / 0.611 | 11.32 / 0.460 | 12.45 / 0.537 |

1 source view, correct rays

2 source views, correct rays

4 source views, correct rays

2 source views, shuffled rays

2 source views, no rays
Number of source views. More source views help. With 4 views the model sees more of the area the target camera is looking at, so the result gets closer to the ground truth, even though I only trained with 2.
Ray conditioning. The target rays clearly matter. Without them the model doesn't know which view to render, so it outputs a blurry average of the scene, and adding more source views doesn't help at all. Shuffled rays are even worse than no rays, because the model confidently renders the wrong viewpoint. Overall the predictions get the layout and colors right but are still quite blurry.
Question. What happens to PSNR, SSIM and sharpness if I add an SSIM term to the MSE loss?
Setup. Loss = MSE + 0.05 × (1 − SSIM). Everything else is the same as the baseline, and both runs were trained for 100 epochs so they can be compared directly. Results are on the validation set with 2 source views.
| Loss | PSNR | SSIM |
|---|---|---|
| MSE | 16.15 | 0.560 |
| MSE + SSIM | 15.95 | 0.582 |
Columns: ground truth, MSE, MSE + SSIM.
Results. SSIM went up and PSNR went down a little, which makes sense since part of the loss now goes toward SSIM instead of pixel error. I expected the images to look sharper, but they actually look smoother. My guess is that SSIM rewards getting local brightness and contrast right, and a smooth image can do that without getting the fine details right. One thing to keep in mind is that the MSE results have a visible blocky pattern that can look like sharpness. This is also only one run with one loss weight.
Total, camera and depth loss.
Ground truth (left) and prediction (right). After overfitting, the prediction is almost identical to the ground truth.
Training settings: the default config, with 8 views, sampling range 32, batch size 16, 1000 epochs, learning rate 6e-4 with 10 warmup epochs, and a frozen DINOv2 ViT-B/14 encoder. Training took about 4 hours on one RTX 5090. The best checkpoint by validation loss was from epoch 876.
Total, camera and depth loss.
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel | 0.146 | 0.158 |
| Depth δ1 | 0.817 | 0.818 |
| Rotation AUC @5 / 15 / 30 | 0.194 / 0.530 / 0.716 | 0.201 / 0.529 / 0.723 |
| Translation AUC @5 / 15 / 30 | 0.004 / 0.071 / 0.189 | 0.010 / 0.076 / 0.193 |
| Joint pose AUC @5 / 15 / 30 | 0.002 / 0.057 / 0.172 | 0.003 / 0.057 / 0.176 |
Errors on held-out scenes. Depth and rotation work reasonably well on new scenes, and the overall room shape usually looks right. Translation is the weak part. The model often gets the direction the camera moved wrong, which also pulls down the joint pose score. It got this almost perfect when overfitting, so it can learn it for one clip but doesn't generalize. One likely reason is that the cameras in a clip don't move much, so the direction is hard to tell. The predicted depth is also too smooth, so thin things like chair and table legs disappear, and flat walls sometimes bend, like in the hallway scene. Val and test results are very close.
Question. Is it better to predict 3D points directly, or to build them from the predicted cameras and depth like the baseline does?
Setup. I added a point head next to the depth head. It predicts a 3D point for every pixel in the first camera's frame, and is trained with an L2 point loss plus the same gradient loss used for depth. Everything else is the same as the baseline, also 1000 epochs. I compare the average distance between predicted and true points in meters.
| Points from | Val error (m) | Test error (m) |
|---|---|---|
| Baseline: cameras + depth | 0.530 | 0.452 |
| Point model: cameras + depth | 0.522 | 0.466 |
| Point model: point head | 0.539 | 0.477 |
Loss curve of the point model.
Ground truth, points from cameras + depth, and points from the point head.
Results. Predicting points directly did not help. The point head was slightly worse than building points from the cameras and depth, but all three are within a few centimeters of each other, which is probably within the noise of a single run. Visually the point clouds look almost the same and have the same blurry furniture. I think the reason is that a point map in the first camera's frame still has to figure out where every camera is, so it runs into the same translation problem. The extra loss also competes with the camera and depth losses. Since I only ran each model once with one loss weight, I can't say for sure that one is better.