check_data.py output
(l3d) ➜ assignment4_f26 git:(main) ✗ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 0.00010755637777037919,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.281559323426336e-05,
"max_ray_reprojection_error_px": 8.446346328128129e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010796334390761331,
"max_ray_reprojection_error_px": 9.860387945082039e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
PSNR = 30.99
SSIM = 0.87
For each image, the first n columns are the source views, the second last column is the ground truth and the last column is the predicted image.
Validation metrics for each of the above experiments are summarized in the table below:
| Source Views | Ray Type | PSNR | SSIM |
|---|---|---|---|
| 1 | Correct | 15.19 | 0.55 |
| 1 | Shuffled | 11.80 | 0.48 |
| 1 | None | 12.46 | 0.52 |
| 2 | Correct | 16.13 | 0.55 |
| 2 | Shuffled | 11.49 | 0.47 |
| 2 | None | 12.44 | 0.52 |
| 4 | Correct | 16.87 | 0.57 |
| 4 | Shuffled | 11.59 | 0.47 |
| 4 | None | 12.46 | 0.53 |
Training settings:
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"batch_size": 4,
"eval_num_frames": 32
},
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true,
"vis_every": 5,
"output_dir": "outputs/lvsm"
},
"num_source_views": 2
}
Most of the original training settings were used. Batch size was reduced to 4 due to memory constraints.
Increasing the number of source views improves novel-view synthesis when the provided target rays are correct. With the correct rays, increasing the number of views from 1 to 4 improves PSNR from 15.19 to 16.87 and SSIM from 0.55 to 0.57. However, the number of source views has little to no effect when the rays are shuffled or replaced with zeros. The ray conditioning ablation shows that the correct rays are essential, the shuffled rays produce the worst results, while the zero rays perform slightly better.
Trained with 1 source view
Same training settings as the previous experiment, but 1 source view.
Metrics
| Source Views | Ray Type | PSNR | SSIM |
|---|---|---|---|
| 1 | Correct | 15.11 | 0.55 |
| 1 | Shuffled | 11.84 | 0.49 |
| 1 | None | 12.46 | 0.53 |
| 2 | Correct | 14.90 | 0.55 |
| 2 | Shuffled | 11.94 | 0.49 |
| 2 | None | 12.41 | 0.53 |
| 4 | Correct | 16.64 | 0.56 |
| 4 | Shuffled | 12.16 | 0.51 |
| 4 | None | 12.43 | 0.54 |
Comparing the two models, training with 2 source views generally improves performance when correct ray conditioning is provided, particularly when evaluated with 2 views. However, the difference is much smaller when evaluated with 1 or 4 views. In contrast the model trained with 1 view performs slightly better when the rays are shuffled, although both models perform a lot worse under incorrect ray conditioning. The zero ray results are nearly identical, which means that the accuracy of the ray conditioning has a greater effect on reconstruction quality than the number of source views used during training.
The difference between the models is relatively small in some settings, particularly at 1 and 4 evaluation views.
The experiment should be repeated with more views as well to see if there is a trend that is observed.
| Metric | Value |
|---|---|
| Depth AbsRel | 0.1492 |
| Depth δ1 | 0.8152 |
| Metric | AUC @ 5° | AUC @ 15° | AUC @ 30° |
|---|---|---|---|
| Rotation AUC | 0.1671 | 0.4771 | 0.6929 |
| Translation AUC | 0.0184 | 0.0709 | 0.1753 |
| Joint Pose AUC | 0.0028 | 0.0378 | 0.1521 |
| Metric | Value |
|---|---|
| Depth AbsRel | 0.1558 |
| Depth δ1 | 0.8052 |
| Metric | AUC @ 5° | AUC @ 15° | AUC @ 30° |
|---|---|---|---|
| Rotation AUC | 0.1781 | 0.4679 | 0.6723 |
| Translation AUC | 0.0151 | 0.0720 | 0.1731 |
| Joint Pose AUC | 0.0028 | 0.0461 | 0.1473 |
Configuration used:
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 8,
"num_workers": 4
},
"model": {
"embed_dim": 384,
"num_heads": 6,
"num_layers": 8,
"predict_points": false,
"encoder_name": "dinov2_vitb14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 1000,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"output_dir": "outputs/vggt",
"overfit_scenes": 1,
"overfit_steps_per_epoch": 25,
"min_lr_ratio": 0.05,
"log_every": 10,
"warmup_epochs": 10,
"overfit_epochs": 40,
"overfit_learning_rate": 0.001
}
}
For held out scenes (validation and testing) the model recovered the geometry of the scene decently well, but made errors in smaller objects, camera alignment and certain boundaries for far away objects. Test and validation metrics are decent, with reasonable generalization, without overfitting. The model produces decent scenes but the predictions are still approximate in ambiguous regions.
For this experiment, the DINO weights were made trainable, and the entire model was trained with the same configurations.
| Metric | Standard VGGT | Unfrozen DINO |
|---|---|---|
| Depth AbsRel | 0.1492 | 0.2848 |
| Depth δ1 | 0.8152 | 0.5798 |
| Metric | AUC @ 5° (Standard / DINO) | AUC @ 15° (Standard / DINO) | AUC @ 30° (Standard / DINO) |
|---|---|---|---|
| Rotation AUC | 0.1671 / 0.1079 | 0.4771 / 0.3174 | 0.6929 / 0.5354 |
| Translation AUC | 0.0184 / 0.0048 | 0.0709 / 0.0310 | 0.1753 / 0.1037 |
| Joint Pose AUC | 0.0028 / 0.0015 | 0.0378 / 0.0166 | 0.1521 / 0.0860 |
| Metric | Standard VGGT | Unfrozen DINO |
|---|---|---|
| Depth AbsRel | 0.1558 | 0.2479 |
| Depth δ1 | 0.8052 | 0.6307 |
| Metric | AUC @ 5° (Standard / DINO) | AUC @ 15° (Standard / DINO) | AUC @ 30° (Standard / DINO) |
|---|---|---|---|
| Rotation AUC | 0.1781 / 0.0962 | 0.4679 / 0.2941 | 0.6723 / 0.4993 |
| Translation AUC | 0.0151 / 0.0015 | 0.0720 / 0.0291 | 0.1731 / 0.1014 |
| Joint Pose AUC | 0.0028 / 0.0000 | 0.0461 / 0.0149 | 0.1473 / 0.0818 |
Unfreezing the DINO backbone resulted in worse performances across every reported metric compared to the standard VGGT with a frozen DINO backbone. This suggests that fine-tuning the DINO backbone under the current setup negatively impacted the learned parameters. This is probably because the model was not trained for long enough and with enough data.