1. Multi-View Data Preprocessing (20 points)

1.2. Depth Unprojection

Scene 036bce3393, unprojection
Scene 036bce3393
Scene 07f5b601ee, unprojection
Scene 07f5b601ee

1.3. Camera and Depth Canonicalization

Scene 036bce3393, canonicalization
Scene 036bce3393
Scene 07f5b601ee, canonicalization
Scene 07f5b601ee

1.7. check_data.py Output

Canonicalization

Cameras and points are expressed relative to the first camera. Camera positions, depths, and points are divided by scene scale. This keeps the first camera at the origin and makes the valid-point centroid distance 1.

World-space ray directions

Pixel directions are computed using the inverse camera intrinsics (K), rotated into the world coordinates using the camera rotation, and normalized to length 1. Each ray’s origin is its camera’s position.

Plücker moment

The Plücker moment is m = o × d, where o is the ray origin and d is its unit direction. It describes the line’s location relative to the coordinate origin.

Full output (five local training samples)
{
  "split": "train",
  "dataset_samples": 5,
  "checked_samples": 5,
  "samples": [
    {
      "sequence_name": "07ff1c45bb_frame_002580",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9826286435127258,
      "max_reprojection_error_px": 0.00010816387657541782,
      "max_ray_reprojection_error_px": 8.648516813991591e-05,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "07f5b601ee_frame_006080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.8545569777488708,
      "max_reprojection_error_px": 0.00011627007188508287,
      "max_ray_reprojection_error_px": 0.00011726274533430114,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "036bce3393_frame_006330",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9630177021026611,
      "max_reprojection_error_px": 9.770392352947965e-05,
      "max_ray_reprojection_error_px": 0.00010235906665911898,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "076c822ecc_frame_001080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.933085560798645,
      "max_reprojection_error_px": 0.00013762763410340995,
      "max_ray_reprojection_error_px": 0.00012943384354002774,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "079a326597_frame_004300",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9994071125984192,
      "max_reprojection_error_px": 8.897326188161969e-05,
      "max_ray_reprojection_error_px": 8.415365300606936e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      1,
      8,
      3,
      224,
      224
    ],
    "depth": [
      1,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      1,
      8,
      1,
      224,
      224
    ],
    "K": [
      1,
      8,
      3,
      3
    ],
    "frame_numbers": [
      1,
      8
    ],
    "T_world_from_camera": [
      1,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      1,
      4,
      4
    ],
    "scene_scale": [
      1
    ],
    "points_world": [
      1,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      1,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      1,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      1,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM (40 points)

2.1. Overfitting

After 2000 training steps on one fixed batch, the final overfit PSNR is 30.86 dB.

val_epoch000.png after 2000 steps.

2.2. Training and Evaluation

Training settings

I used the provided configs/lvsm.json without changes and completed all 100 epochs (5600 training steps).

Training curves

Training and validation curves, from log.jsonl.

Ablations

Evaluated with best.pt (epoch 99) on the validation split. The model was trained with 2 source views only, so 1 and 4 are source counts it never saw in training.

Source views Target rays PSNR (dB) SSIM Images
1 Correct 15.20 0.5586 196
1 Shuffled 11.80 0.4879 196
1 None 12.46 0.5280 196
2 Correct 16.13 0.5602 168
2 Shuffled 11.50 0.4724 168
2 None 12.44 0.5259 168
4 Correct 16.86 0.5753 112
4 Shuffled 11.59 0.4731 112
4 None 12.45 0.5357 112

Training visualizations

Validation scene during training.

Epoch 0
Epoch 50
Epoch 95

Evaluation visualizations

1 source, correct rays
1 source, shuffled rays
1 source, no rays
2 sources, correct rays
2 sources, shuffled rays
2 sources, no rays
4 sources, correct rays
4 sources, shuffled rays
4 sources, no rays

Discussion

Source views. With 1, 2 and 4 sources, PSNR goes from 15.20 to 16.13 to 16.86 dB. So more views help even at source counts the model never trained on.

Ray conditioning. Correct rays beat shuffled rays by 3.4 to 5.3 dB and no rays by 2.7 to 4.4 dB. The gap grows with more sources. Surprizingly, no rays scores better than shuffled rays. It may be that misleading camera rays hurt more than omitting.

2.3. Bonus: Training with One Source View

Question. How does training with one source view instead of two affect novel-view synthesis with 1, 2 and 4 sources at evaluation?

Settings. I trained a new model from scratch for 100 epochs (5600 steps), changing only the training source count from 2 to 1 in the baseline configuration. This gives 7 target views per training clip instead of 6. Each model uses its best validation-PSNR checkpoint.

Results. Both models use correct target rays and the same 28 validation scenes. At each evaluation source count, the source images, target images and normalization are identical between models.

Sources at evaluation Trained with 2: PSNR Trained with 1: PSNR Trained with 2: SSIM Trained with 1: SSIM
1 15.20 15.11 0.5586 0.5590
2 16.13 14.91 0.5602 0.5527
4 16.86 14.64 0.5753 0.5609
Validation PSNR and SSIM with identical evaluation views for the two models.
Validation PSNR and SSIM with identical evaluation views for the two models.
1 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.
1 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.
2 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.
2 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.
4 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.
4 evaluation source views. Columns: sources, GT, two-source-trained prediction, one-source-trained prediction.

Discussion. Training with one source changes PSNR by -0.09 dB at 1 source, -1.22 dB at 2 sources, -2.23 dB at 4 sources. Almost no change at 1 source evaluation, gains in performance appear to increase with more eval sources. This suggests that training with multiple sources helped the model learn to combine information across images. Qualitatively, the visualizations have stronger figures and sharper edges, resulting in clearer images (although they are still very blurred).

3. Multi-View Geometry with Mini-VGGT (40 points)

3.1. Overfitting

Overfit total, camera and depth losses.

GT and predicted geometry

Scene 036bce3393 — Ground truth
Scene 036bce3393 — Ground truth
Scene 036bce3393 — Prediction
Scene 036bce3393 — Prediction

3.2. Training and Evaluation

Training settings

I used the provided configs/vggt.json with two changes:

Setting Default Used
Training epochs 1000 680
Data-loading workers 8 2

The epoch count was reduced to fit the planned eight-hour training budget. All other model and training settings were unchanged, including the required 8 views and sampling range of 32.

Training curves

Training and validation total, camera and depth losses.

Validation and test metrics

I evaluated best.pt (epoch 680, selected by validation loss) on all 28 validation and 28 test scenes, using 8 views from the first 32 prepared frames, 224px images and seed 825, with no batch limit.

Metric Validation Test
Depth AbsRel ↓ 0.1646 0.1675
Depth δ1 ↑ 0.8059 0.7795
Rotation AUC @ 5° ↑ 0.1686 0.1918
Rotation AUC @ 15° ↑ 0.4816 0.4907
Rotation AUC @ 30° ↑ 0.6768 0.6952
Translation AUC @ 5° ↑ 0.0196 0.0069
Translation AUC @ 15° ↑ 0.0731 0.0520
Translation AUC @ 30° ↑ 0.1662 0.1533
Joint pose AUC @ 5° ↑ 0.0056 0.0013
Joint pose AUC @ 15° ↑ 0.0516 0.0325
Joint pose AUC @ 30° ↑ 0.1408 0.1337

GT and predicted geometry: four validation scenes

Scene 09c1414f1b — Ground truth
Scene 09c1414f1b — Ground truth
Scene 09c1414f1b — Prediction
Scene 09c1414f1b — Prediction
Scene 21d970d8de — Ground truth
Scene 21d970d8de — Ground truth
Scene 21d970d8de — Prediction
Scene 21d970d8de — Prediction
Scene 280b83fcf3 — Ground truth
Scene 280b83fcf3 — Ground truth
Scene 280b83fcf3 — Prediction
Scene 280b83fcf3 — Prediction
Scene 2b1dc6d6a5 — Ground truth
Scene 2b1dc6d6a5 — Ground truth
Scene 2b1dc6d6a5 — Prediction
Scene 2b1dc6d6a5 — Prediction

Held-out errors

The predicted depth smooths object boundaries and thin structures, especially in thin detailed structures like the dining chairs in scene 09c1414f1b. The clouds retain the broad scene layout, but surfaces look warped, spread out, or phasing through itself in scenes 280b83fcf3 and 2b1dc6d6a5. Camera-motion direction is still off. Average angular error is 43.15° on validation and 42.54° on test, compared with rotation errors of 12.89° and 10.09°. These camera errors can prevent points from different views from lining up, explaining some of the erroneous artifacts in the surfaces.

3.3. Bonus: Number of Input Views

Question. Does adding context views improve camera and depth predictions for the same two images, and what does it cost in runtime and memory?

Settings. I reused the Q3.2 best checkpoint (epoch 680) without further training. For each of the 28 validation scenes, I loaded the same eight frames from the first 32 prepared frames (224px, seed 825), then supplied nested sets of 2, 4 and 8 views. All settings score only the same first two views and their camera pair.

Results. Accuracy below is on the common two views, so it differs from Q3.2’s metrics over all eight views.

Input views Depth AbsRel ↓ Depth δ1 ↑ Rotation AUC @ 15° ↑ Translation AUC @ 15° ↑ Joint pose AUC @ 15° ↑ Forward ms Peak MiB
2 0.1828 0.6845 0.5048 0.0929 0.0786 36.9 709
4 0.1613 0.8132 0.5548 0.0738 0.0690 36.6 728
8 0.1563 0.8298 0.4833 0.0738 0.0643 36.3 771
Accuracy on the same two views, plus forward time and peak allocated memory as input context grows.
Accuracy on the same two views, plus forward time and peak allocated memory as input context grows.
Depth of the same first validation view with 2, 4 and 8 inputs. GT and predictions share depth colors; white pixels are GT-invalid.
Depth of the same first validation view with 2, 4 and 8 inputs. GT and predictions share depth colors; white pixels are GT-invalid.

Discussion. From 2 to 8 inputs, AbsRel changes from 0.1828 to 0.1563, δ1 from 0.6845 to 0.8298, and joint pose AUC @ 15° from 0.0786 to 0.0643. The lower depth error with more context is consistent with extra views helping constrain scene geometry. The depth benefit does not extend to joint pose AUC at 15°, which declines here. Measured latency stays around 36–37 ms across conditions; the small decrease is not enough evidence of a speedup from this ten-forward, single-scene benchmark. Peak allocated memory rises from 709 to 771 MiB. This study uses one fixed checkpoint and only one pose pair per scene, so small AUC differences should be interpreted cautiously.