



sequence valid depth max reprojection (px) max ray reprojection (px) centroid distance e050c15a8d_frame_007080 0.9779 1.13e-04 1.09e-04 1.0 5f99900f09_frame_002820 0.9903 1.23e-04 1.11e-04 1.0 47b37eb6f9_frame_002720 0.9610 9.78e-05 8.90e-05 1.0 4c5c60fa76_frame_006690 0.7517 1.17e-04 1.07e-04 1.0
PSNR = 30.91 dB
"data_overrides": {
"batch_size": 4,
"eval_num_frames": 32
},
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true,
"vis_every": 5,
"output_dir": "outputs/lvsm"
},
"num_source_views": 2
| Source views | Conditioning | PSNR | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.558 |
| 1 | shuffled | 11.81 | 0.488 |
| 1 | none | 12.47 | 0.528 |
| 2 | correct | 16.13 | 0.560 |
| 2 | shuffled | 11.50 | 0.473 |
| 2 | none | 12.45 | 0.527 |
| 4 | correct | 16.86 | 0.575 |
| 4 | shuffled | 11.60 | 0.473 |
| 4 | none | 12.48 | 0.537 |
Training visualization, epoch 95. Each row: sources | ground truth | prediction.
Correct rays. Left: 1 source, middle: 2 sources, right: 4 sources.
2 sources. Left: correct, middle: shuffled, right: none.
More sources views seem to help, leading to an increase in PSNR. More sources cover more of the scene, so each target pixel is more likely to be seen by at least one of them.
Without the correct rays, the PSNR drops. Also, it seems that having no rays at all is better than having the wrong rays (at least, for PSNR). With the wrong rays, the model may be predicting the right pixel in the wrong location, leading to a worse result for PSNR (but better, visually).
Since the results look blurry either way, I experimented with adding a penalty for lower SSIM (weight of 0.1) since that also computes local statistics.
| Loss | 1 source PSNR / SSIM | 2 sources PSNR / SSIM | 4 sources PSNR / SSIM |
|---|---|---|---|
| MSE | 15.19 / 0.558 | 16.13 / 0.560 | 16.86 / 0.575 |
| MSE + SSIM | 14.88 / 0.575 | 15.54 / 0.578 | 15.97 / 0.591 |
4 sources. Left: MSE, right: MSE + SSIM.
SSIM goes up a little and PSNR goes down. The patch edges are gone, but the predictions are smoother. My hypothesis is that a blurry image with the right colors in the right places already gets a decent SSIM. A sharp edge in the wrong place would lower it, and so the model has no reason to predict sharp edges. The train PSNR is about as low as val, so the model is underfitting.
Ground truth

Prediction
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 16,
"num_workers": 8
},
"model": {
"embed_dim": 384,
"num_heads": 6,
"num_layers": 8,
"predict_points": false,
"encoder_name": "dinov2_vitb14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 1000,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"min_lr_ratio": 0.05,
"warmup_epochs": 10
}
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel | 0.141 | 0.158 |
| Depth δ1 | 0.835 | 0.806 |
| Rotation AUC @5 / @15 / @30 | 0.228 / 0.556 / 0.716 | 0.199 / 0.526 / 0.734 |
| Translation AUC @5 / @15 / @30 | 0.014 / 0.083 / 0.201 | 0.012 / 0.074 / 0.192 |
| Joint pose AUC @5 / @15 / @30 | 0.004 / 0.067 / 0.176 | 0.002 / 0.050 / 0.174 |
09c1414f1b ground truth

09c1414f1b prediction
21d970d8de ground truth

21d970d8de prediction
280b83fcf3 ground truth

280b83fcf3 prediction
2b1dc6d6a5 ground truth

2b1dc6d6a5 prediction
The clouds appeared smeared along the viewing direction. The views that would have pinned those aspects down don't exist. The predicted cameras also seemed to be bunched closer than the ground truth cameras.
The model is trained with 8 views. How does it do with fewer or more views at test time?
| Views | AbsRel | δ1 | Rotation AUC @5 / @15 / @30 | Translation AUC @5 / @15 / @30 | Pose AUC @5 / @15 / @30 | Pose pairs | Wall time (s) |
|---|---|---|---|---|---|---|---|
| 2 | 0.116 | 0.861 | 0.264 / 0.590 / 0.735 | 0.000 / 0.095 / 0.213 | 0.000 / 0.074 / 0.185 | 28 | 3.5 |
| 4 | 0.127 | 0.851 | 0.214 / 0.537 / 0.699 | 0.023 / 0.068 / 0.172 | 0.010 / 0.055 / 0.148 | 168 | 3.7 |
| 8 | 0.141 | 0.834 | 0.228 / 0.556 / 0.716 | 0.014 / 0.083 / 0.201 | 0.004 / 0.067 / 0.176 | 784 | 4.8 |
| 16 | 0.174 | 0.782 | 0.206 / 0.504 / 0.675 | 0.011 / 0.066 / 0.171 | 0.003 / 0.048 / 0.145 | 3360 | 5.0 |
| 32 | 0.192 | 0.758 | 0.184 / 0.490 / 0.676 | 0.013 / 0.070 / 0.177 | 0.003 / 0.048 / 0.149 | 13888 | 7.4 |
The depth gets worse with more views. Some of this is because more views cover more of the clip, but the jump above 8 views is larger, so the model also seems to do worse with more views than it was trained with. Presumably the global attention now sees a lot more tokens than in training. Pose is best around 8 views (the 2-view result is only 28 pairs, so it is noisy). The runtime grows with the number of views. Training with a random number of views should help.