Andrew ID: yunxianm
I resize the intrinsics using the same pixel-center convention as the image resize: x_out + 0.5 = sx * (x_in + 0.5), and likewise for y. Thus the focal lengths scale by sx, sy, while the principal point also receives the half-pixel offset. The provided loader has already adjusted the intrinsics for the center crop.
For a pixel p = [u, v, 1], I compute X_camera = depth * inverse(K) * p, then apply the camera-to-world rotation and translation. I do not normalize this camera-space vector before multiplying by depth: the measurements are optical-axis z-depth.
Canonicalization applies inverse(T_0) * T_i to every camera. The shared scene scale is the distance from camera 0 to the mean valid 3D point. Dividing camera translations, depths and points by this scale puts the centroid one unit from the origin. This gives scenes a common coordinate frame and scale without changing the relative geometry.
Each ray starts at its camera center. Its direction is normalize(R * inverse(K) * p), expressed in the canonical world frame. The Plücker representation is [d, o × d]. The moment encodes the ray's offset from the origin; moving o along the same ray leaves the moment unchanged because d × d = 0.
The following pairs use the same sampled frames in each scene. The left image is in the original world frame; the right is canonicalized. Each view displays at most 10000 valid points. The canonicalized screenshots show camera 0 at the origin and a valid-point centroid distance of 1.000.
| Scene | Original world frame | Canonical frame |
|---|---|---|
| 036bce3393 | ![]() |
![]() |
| 076c822ecc | ![]() |
![]() |
| 079a326597 | ![]() |
![]() |
| 07f5b601ee | ![]() |
![]() |
| 07ff1c45bb | ![]() |
![]() |
check_data.py passed on four training samples, each with eight 224x224 views. The largest point reprojection error was approximately 0.000123 pixels, and the largest ray reprojection error was approximately 0.000111 pixels, both below 0.01 pixels. All four centroid distances rounded to 1.000000. The original output is shown below and saved in check_data.json.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Training samples draw frames in random order from a temporal window and run the geometry functions above. Validation and test use the provided prepared frames, keep frame 0 as the reference, and bypass these functions. The models use the first 32 prepared frames for evaluation.
I concatenate each source image in [-1, 1] with its six Plücker channels, then split it into 8x8 patches. Target tokens contain rays only. Each target gets its own sequence of source tokens followed by target tokens, with a shared 2D positional embedding. After eight transformer blocks, I decode only the target tokens into RGB. The loss is MSE over every target pixel.
I trained on one fixed batch for 2000 steps. Final PSNR on that batch was 30.92 dB, with MSE 8.647e-4 and SSIM 0.8702. The model reproduces the room layout, whiteboard and chairs. Some fine-texture errors and patch boundaries remain visible.
Columns: two source views, target ground truth, prediction.

I trained for 100 epochs on 224 scenes with eight views per sample: two sources and six targets. Images are 224x224, the sampling range is 32, and batch size is 4. I used AdamW with learning rate 2e-4, weight decay 0.05, 200 warmup steps, gradient clipping at 1.0, and bfloat16 autocast. The best validation PSNR occurred at epoch 100.

Validation PSNR was still improving near epoch 100. More epochs may help.
Validation example saved during training at epoch 96. Columns are the two source images, target GT, and prediction; each row uses a different target.

The following evaluates the best checkpoint on all 28 validation scenes. Each scene has eight selected views from the first 32 prepared frames. Using one, two or four sources leaves seven, six or four target views, respectively. shuffled substitutes another target's rays, while none sets the target rays to zero.
| Source views | Target rays | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.20 | 0.5580 |
| 1 | shuffled | 11.81 | 0.4872 |
| 1 | none | 12.47 | 0.5276 |
| 2 | correct | 16.13 | 0.5600 |
| 2 | shuffled | 11.49 | 0.4718 |
| 2 | none | 12.45 | 0.5273 |
| 4 | correct | 16.87 | 0.5751 |
| 4 | shuffled | 11.59 | 0.4727 |
| 4 | none | 12.45 | 0.5374 |
Full metrics: lvsm_val_metrics.json.
Columns in each image are the source views, target GT, and prediction.
| Sources | Evaluation example with correct rays |
|---|---|
| 1 | ![]() |
| 2 | ![]() |
| 4 | ![]() |
More sources improve PSNR from 15.20 to 16.87 dB and SSIM from 0.5580 to 0.5751. Additional views reveal more of the room and reduce ambiguity about visible surfaces. The model still mainly predicts coarse color and layout: furniture edges are blurred, and patch boundaries remain visible. Pixel MSE favors averaging when the model cannot resolve the exact appearance.
| Two-source setting | Evaluation example |
|---|---|
| Shuffled rays | ![]() |
| Zero rays | ![]() |
With two sources, shuffling the rays reduces PSNR from 16.13 to 11.49 dB. Bright windows and dark furniture move to incorrect image locations, so the model relies on target rays to pick the viewpoint. Zero rays produce nearly identical predictions for different targets and reach 12.45 dB. Adding sources does not recover the correct viewpoint when its rays are missing or mismatched.
To reduce the blur in the MSE baseline, I trained a second model with loss = MSE + 0.1 L1, keeping the architecture, seed 825, data sampling, optimizer and 100-epoch schedule unchanged. Each loss was trained once; both runs selected epoch 100 by validation PSNR. I evaluated the same 28 validation scenes with correct target rays.
| Source views | MSE PSNR (dB) | MSE + 0.1 L1 PSNR (dB) | MSE SSIM | MSE + 0.1 L1 SSIM |
|---|---|---|---|---|
| 1 | 15.20 | 15.25 | 0.5580 | 0.5590 |
| 2 | 16.13 | 16.23 | 0.5600 | 0.5619 |
| 4 | 16.87 | 17.00 | 0.5751 | 0.5776 |


The same two-source example is shown below. Columns are sources, GT and prediction.
| MSE | MSE + 0.1 L1 |
|---|---|
![]() |
![]() |
L1 improves PSNR by 0.05 to 0.12 dB across the three source counts, with a small SSIM increase. The chair and bookshelf remain blurred, and the patch grid is still visible.
L1 gives small residuals more weight relative to MSE, which may explain the small gains without a clear improvement in sharpness. The complete nine-setting L1 evaluation is in lvsm_l1_val_metrics.json.
I alternate attention within each frame and across all frames, then concatenate the last frame and global features. The camera head predicts translation, quaternion and field of view. The depth head adds RGB refinement before exponentiating depth, and predicts confidence as 1 + exp(raw_confidence).
I used the provided frozen DINOv2 ViT-B/14 encoder and fitted one eight-view training clip for 40 epochs, with 25 steps per epoch. The total loss fell from 5.4835 to -0.1619. Camera loss fell from 0.3424 to 0.002483, and depth loss from 3.7715 to -0.1743. The depth objective can become negative because it includes the confidence term -0.2 log(confidence).
Depth AbsRel decreased from 0.6186 to 0.03761, and δ1 increased from 20.81% to 95.36% on the fixed clip. The predicted camera path and main surfaces align closely with GT. The predicted depth is smoother around object edges and small clutter.

| Ground truth | Predicted cameras and depth |
|---|---|
![]() |
![]() |
I trained for 1000 epochs on the 224 training scenes with eight views per sample and sampling_range=32. I kept the default model, batch size 16, AdamW learning rate 6e-4, weight decay 0.01, and a 10-epoch warmup followed by cosine decay. The encoder stayed frozen. The lowest validation loss selected the checkpoint from epoch 838.

Depth loss drops sharply early in training, while camera loss continues to decrease over many more epochs. I evaluated all 28 scenes in each held-out split using eight views from the first 32 prepared frames, 224x224 images, and seed 825.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.1436 | 0.1555 |
| Depth δ1 | 0.8318 | 0.7870 |
| Rotation AUC @ 5° | 0.1921 | 0.1946 |
| Rotation AUC @ 15° | 0.5127 | 0.5025 |
| Rotation AUC @ 30° | 0.6855 | 0.6980 |
| Translation AUC @ 5° | 0.0102 | 0.0046 |
| Translation AUC @ 15° | 0.0700 | 0.0435 |
| Translation AUC @ 30° | 0.1727 | 0.1464 |
| Joint pose AUC @ 5° | 0.0028 | 0.0015 |
| Joint pose AUC @ 15° | 0.0496 | 0.0338 |
| Joint pose AUC @ 30° | 0.1461 | 0.1316 |
Full metrics: validation, test.
| Validation scene | Ground truth | Predicted cameras and depth |
|---|---|---|
| 09c1414f1b | ![]() |
![]() |
| 21d970d8de | ![]() |
![]() |
| 280b83fcf3 | ![]() |
![]() |
| 2b1dc6d6a5 | ![]() |
![]() |
The model captures the broad room layout, but the dining-room chairs and hallway boundaries become smooth and thick. Misaligned views spread surfaces into overlapping layers. Rotation is more accurate than translation direction: on test scenes, their 30° AUCs are 0.6980 and 0.1464, respectively. The low joint pose AUC follows the weaker translation estimate. Test depth is also worse than validation, with δ1 falling from 0.8318 to 0.7870.
For this bonus I only changed the number of input views at evaluation time, using the epoch-838 checkpoint on all 28 validation scenes. I took nested subsets of 2, 4 and 8 views and kept the same scene scale. I score only the first two views in every setting, so these numbers should not be compared directly with Section 3.2.
| Input views | Depth AbsRel | Depth δ1 | Rotation AUC @ 15° | Translation AUC @ 15° | Joint pose AUC @ 15° | Time/scene (ms) | Peak GPU memory (MiB) |
|---|---|---|---|---|---|---|---|
| 2 | 0.1730 | 0.7222 | 0.5524 | 0.0810 | 0.0595 | 53.39 | 712.2 |
| 4 | 0.1437 | 0.8138 | 0.5310 | 0.0738 | 0.0667 | 53.60 | 730.4 |
| 8 | 0.1325 | 0.8517 | 0.5143 | 0.0929 | 0.0857 | 52.82 | 771.2 |
Timing: one A100, batch size 1, bf16, after one warmup per view count. The synchronized prediction call includes RGB transfer to the GPU.

The first four validation scenes are shown below. Each row uses a common depth color scale across GT and predictions; white GT pixels have invalid depth.

Depth improves consistently with more context: AbsRel drops from 0.1730 to 0.1325 and δ1 rises from 0.7222 to 0.8517. The first scene's background depth and the hallway's depth progression move closer to GT. Chair legs and other thin structures are still smoothed out. Extra views provide additional geometric evidence, and eight views also match the training setting.
Pose behavior is less consistent. Joint AUC at 15° improves, but rotation AUC decreases and translation AUC is not monotonic. More context helps depth more consistently than camera orientation. With only 28 camera pairs, I would not use the small pose AUC differences alone to rank the settings; more samples are needed to assess how stable they are. Peak memory increases from 712.2 to 771.2 MiB. Measured prediction times stay near 53 ms across all three view counts. A possible explanation is that batch size 1 leaves the GPU underused and fixed transfer and launch costs mask the extra computation.
Full results and sampled frame indices: vggt_view_metrics.json.