1. Multi-View Data Preprocessing (20 points)
1.2. Depth Unprojection
1.3. Camera and Depth Canonicalization
1.7. check_data.py Output
Canonicalization
Cameras and points are expressed relative to the first camera. Camera positions, depths, and points are divided by scene scale. This keeps the first camera at the origin and makes the valid-point centroid distance 1.
World-space ray directions
Pixel directions are computed using the inverse camera intrinsics (K), rotated into the world coordinates using the camera rotation, and normalized to length 1. Each ray’s origin is its camera’s position.
Plücker moment
The Plücker moment is m = o × d, where o is
the ray origin and d is its unit direction. It describes
the line’s location relative to the coordinate origin.
Full output (five local training samples)
{
"split": "train",
"dataset_samples": 5,
"checked_samples": 5,
"samples": [
{
"sequence_name": "07ff1c45bb_frame_002580",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9826286435127258,
"max_reprojection_error_px": 0.00010816387657541782,
"max_ray_reprojection_error_px": 8.648516813991591e-05,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "07f5b601ee_frame_006080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.8545569777488708,
"max_reprojection_error_px": 0.00011627007188508287,
"max_ray_reprojection_error_px": 0.00011726274533430114,
"centroid_distance": 1.0
},
{
"sequence_name": "036bce3393_frame_006330",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9630177021026611,
"max_reprojection_error_px": 9.770392352947965e-05,
"max_ray_reprojection_error_px": 0.00010235906665911898,
"centroid_distance": 1.0
},
{
"sequence_name": "076c822ecc_frame_001080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.933085560798645,
"max_reprojection_error_px": 0.00013762763410340995,
"max_ray_reprojection_error_px": 0.00012943384354002774,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "079a326597_frame_004300",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9994071125984192,
"max_reprojection_error_px": 8.897326188161969e-05,
"max_ray_reprojection_error_px": 8.415365300606936e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
1,
8,
3,
224,
224
],
"depth": [
1,
8,
1,
224,
224
],
"depth_valid": [
1,
8,
1,
224,
224
],
"K": [
1,
8,
3,
3
],
"frame_numbers": [
1,
8
],
"T_world_from_camera": [
1,
8,
4,
4
],
"original_world_from_reference": [
1,
4,
4
],
"scene_scale": [
1
],
"points_world": [
1,
8,
3,
224,
224
],
"ray_origins": [
1,
8,
3,
224,
224
],
"ray_directions": [
1,
8,
3,
224,
224
],
"plucker": [
1,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM (40 points)
2.1. Overfitting
After 2000 training steps on one fixed batch, the final overfit PSNR is 30.86 dB.
2.2. Training and Evaluation
Training settings
I used the provided configs/lvsm.json without changes
and completed all 100 epochs (5600 training steps).
Training curves
Ablations
Evaluated with best.pt (epoch 99) on the validation
split. The model was trained with 2 source views only, so 1 and 4 are
source counts it never saw in training.
| Source views | Target rays | PSNR (dB) | SSIM | Images |
|---|---|---|---|---|
| 1 | Correct | 15.20 | 0.5586 | 196 |
| 1 | Shuffled | 11.80 | 0.4879 | 196 |
| 1 | None | 12.46 | 0.5280 | 196 |
| 2 | Correct | 16.13 | 0.5602 | 168 |
| 2 | Shuffled | 11.50 | 0.4724 | 168 |
| 2 | None | 12.44 | 0.5259 | 168 |
| 4 | Correct | 16.86 | 0.5753 | 112 |
| 4 | Shuffled | 11.59 | 0.4731 | 112 |
| 4 | None | 12.45 | 0.5357 | 112 |
Training visualizations
Validation scene during training.
Evaluation visualizations
Discussion
Source views. With 1, 2 and 4 sources, PSNR goes from 15.20 to 16.13 to 16.86 dB. So more views help even at source counts the model never trained on.
Ray conditioning. Correct rays beat shuffled rays by 3.4 to 5.3 dB and no rays by 2.7 to 4.4 dB. The gap grows with more sources. Surprizingly, no rays scores better than shuffled rays. It may be that misleading camera rays hurt more than omitting.
2.3. Bonus: Training with One Source View
Question. How does training with one source view instead of two affect novel-view synthesis with 1, 2 and 4 sources at evaluation?
Settings. I trained a new model from scratch for 100 epochs (5600 steps), changing only the training source count from 2 to 1 in the baseline configuration. This gives 7 target views per training clip instead of 6. Each model uses its best validation-PSNR checkpoint.
Results. Both models use correct target rays and the same 28 validation scenes. At each evaluation source count, the source images, target images and normalization are identical between models.
| Sources at evaluation | Trained with 2: PSNR | Trained with 1: PSNR | Trained with 2: SSIM | Trained with 1: SSIM |
|---|---|---|---|---|
| 1 | 15.20 | 15.11 | 0.5586 | 0.5590 |
| 2 | 16.13 | 14.91 | 0.5602 | 0.5527 |
| 4 | 16.86 | 14.64 | 0.5753 | 0.5609 |
Discussion. Training with one source changes PSNR by -0.09 dB at 1 source, -1.22 dB at 2 sources, -2.23 dB at 4 sources. Almost no change at 1 source evaluation, gains in performance appear to increase with more eval sources. This suggests that training with multiple sources helped the model learn to combine information across images. Qualitatively, the visualizations have stronger figures and sharper edges, resulting in clearer images (although they are still very blurred).
3. Multi-View Geometry with Mini-VGGT (40 points)
3.1. Overfitting
GT and predicted geometry
3.2. Training and Evaluation
Training settings
I used the provided configs/vggt.json with two
changes:
| Setting | Default | Used |
|---|---|---|
| Training epochs | 1000 | 680 |
| Data-loading workers | 8 | 2 |
The epoch count was reduced to fit the planned eight-hour training budget. All other model and training settings were unchanged, including the required 8 views and sampling range of 32.
Training curves
Validation and test metrics
I evaluated best.pt (epoch 680, selected by validation
loss) on all 28 validation and 28 test scenes, using 8 views from the
first 32 prepared frames, 224px images and seed 825, with no batch
limit.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1646 | 0.1675 |
| Depth δ1 ↑ | 0.8059 | 0.7795 |
| Rotation AUC @ 5° ↑ | 0.1686 | 0.1918 |
| Rotation AUC @ 15° ↑ | 0.4816 | 0.4907 |
| Rotation AUC @ 30° ↑ | 0.6768 | 0.6952 |
| Translation AUC @ 5° ↑ | 0.0196 | 0.0069 |
| Translation AUC @ 15° ↑ | 0.0731 | 0.0520 |
| Translation AUC @ 30° ↑ | 0.1662 | 0.1533 |
| Joint pose AUC @ 5° ↑ | 0.0056 | 0.0013 |
| Joint pose AUC @ 15° ↑ | 0.0516 | 0.0325 |
| Joint pose AUC @ 30° ↑ | 0.1408 | 0.1337 |
GT and predicted geometry: four validation scenes
Held-out errors
The predicted depth smooths object boundaries and thin structures, especially in thin detailed structures like the dining chairs in scene 09c1414f1b. The clouds retain the broad scene layout, but surfaces look warped, spread out, or phasing through itself in scenes 280b83fcf3 and 2b1dc6d6a5. Camera-motion direction is still off. Average angular error is 43.15° on validation and 42.54° on test, compared with rotation errors of 12.89° and 10.09°. These camera errors can prevent points from different views from lining up, explaining some of the erroneous artifacts in the surfaces.
3.3. Bonus: Number of Input Views
Question. Does adding context views improve camera and depth predictions for the same two images, and what does it cost in runtime and memory?
Settings. I reused the Q3.2 best checkpoint (epoch 680) without further training. For each of the 28 validation scenes, I loaded the same eight frames from the first 32 prepared frames (224px, seed 825), then supplied nested sets of 2, 4 and 8 views. All settings score only the same first two views and their camera pair.
Results. Accuracy below is on the common two views, so it differs from Q3.2’s metrics over all eight views.
| Input views | Depth AbsRel ↓ | Depth δ1 ↑ | Rotation AUC @ 15° ↑ | Translation AUC @ 15° ↑ | Joint pose AUC @ 15° ↑ | Forward ms | Peak MiB |
|---|---|---|---|---|---|---|---|
| 2 | 0.1828 | 0.6845 | 0.5048 | 0.0929 | 0.0786 | 36.9 | 709 |
| 4 | 0.1613 | 0.8132 | 0.5548 | 0.0738 | 0.0690 | 36.6 | 728 |
| 8 | 0.1563 | 0.8298 | 0.4833 | 0.0738 | 0.0643 | 36.3 | 771 |
Discussion. From 2 to 8 inputs, AbsRel changes from 0.1828 to 0.1563, δ1 from 0.6845 to 0.8298, and joint pose AUC @ 15° from 0.0786 to 0.0643. The lower depth error with more context is consistent with extra views helping constrain scene geometry. The depth benefit does not extend to joint pose AUC at 15°, which declines here. Measured latency stays around 36–37 ms across conditions; the small decrease is not enough evidence of a speedup from this ten-forward, single-scene benchmark. Peak allocated memory rises from 709 to 771 MiB. This study uses one fixed checkpoint and only one pose pair per scene, so small AUC differences should be interpreted cautiously.