Assignment 4

1. Multi-View Data Preprocessing

1.2. Depth Unprojection


Unprojection View 1


Unprojection View 2


Unprojection View 3

1.3. Camera and Depth Canonicalization


Canonical View 1


Canonical View 2


Canonical View 3

1.7. Final Check

check_data.py output

(l3d) ➜ assignment4_f26 git:(main) ✗ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 { "split": "train", "dataset_samples": 224, "checked_samples": 4, "samples": [ { "sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 0.00010755637777037919, "centroid_distance": 1.0 }, { "sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 1.0 }, { "sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.281559323426336e-05, "max_ray_reprojection_error_px": 8.446346328128129e-05, "centroid_distance": 1.0 }, { "sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00010796334390761331, "max_ray_reprojection_error_px": 9.860387945082039e-05, "centroid_distance": 1.0 } ], "batch_shapes": { "rgb": [ 2, 8, 3, 224, 224 ], "depth": [ 2, 8, 1, 224, 224 ], "depth_valid": [ 2, 8, 1, 224, 224 ], "K": [ 2, 8, 3, 3 ], "frame_numbers": [ 2, 8 ], "T_world_from_camera": [ 2, 8, 4, 4 ], "original_world_from_reference": [ 2, 4, 4 ], "scene_scale": [ 2 ], "points_world": [ 2, 8, 3, 224, 224 ], "ray_origins": [ 2, 8, 3, 224, 224 ], "ray_directions": [ 2, 8, 3, 224, 224 ], "plucker": [ 2, 8, 6, 224, 224 ] } }

2. Novel-View Synthesis with Mini-LVSM

2.1. Overfitting


Output after overfitting LVSM

PSNR = 30.99 SSIM = 0.87

2.2. Training and Evaluation


Training curve LVSM

For each image, the first n columns are the source views, the second last column is the ground truth and the last column is the predicted image.

Training Progress


Eval output after 1 epochs

Eval output after 45 epochs

Eval output after 95 epochs

Validation

Source View: 1


(1, correct) 1 Source View, True Target Rays

(1, shuffled) 1 Source View, Target Rays of Different View

(1, none) 1 Source View, All-zero Rays

Source View: 2


(2, correct) 2 Source View, True Target Rays

(2, shuffled) 2 Source View, Target Rays of Different View

(2, none) 2 Source View, All-zero Rays

Source View: 4


(4, correct) 4 Source View, True Target Rays

(4, shuffled) 4 Source View, Target Rays of Different View

(4, none) 4 Source View, All-zero Rays

Validation metrics for each of the above experiments are summarized in the table below:

Source Views Ray Type PSNR SSIM
1 Correct 15.19 0.55
1 Shuffled 11.80 0.48
1 None 12.46 0.52
2 Correct 16.13 0.55
2 Shuffled 11.49 0.47
2 None 12.44 0.52
4 Correct 16.87 0.57
4 Shuffled 11.59 0.47
4 None 12.46 0.53

Training settings:

{ "data_config": "configs/scannetpp/iphone/data.json", "data_overrides": { "batch_size": 4, "eval_num_frames": 32 }, "model": { "patch_size": 8, "d_model": 384, "n_layers": 8, "head_dim": 64, "mlp_ratio": 4.0, "use_qk_norm": true }, "loss": {}, "training": { "epochs": 100, "learning_rate": 0.0002, "weight_decay": 0.05, "betas": [0.9, 0.95], "warmup_steps": 200, "grad_clip": 1.0, "amp": true, "vis_every": 5, "output_dir": "outputs/lvsm" }, "num_source_views": 2 }

Most of the original training settings were used. Batch size was reduced to 4 due to memory constraints.


Increasing the number of source views improves novel-view synthesis when the provided target rays are correct. With the correct rays, increasing the number of views from 1 to 4 improves PSNR from 15.19 to 16.87 and SSIM from 0.55 to 0.57. However, the number of source views has little to no effect when the rays are shuffled or replaced with zeros. The ray conditioning ablation shows that the correct rays are essential, the shuffled rays produce the worst results, while the zero rays perform slightly better.


2.3. Bonus: Analysis and Exploration

Training view count: Train with a different number of source views, or a random number per step. How does this change evaluation with 1, 2 and 4 source views?

Trained with 1 source view

Same training settings as the previous experiment, but 1 source view.


Training Progress


Eval output after 1 epochs

Eval output after 45 epochs

Eval output after 95 epochs

Validation

Source View: 1


(1, correct) 1 Source View, True Target Rays

(1, shuffled) 1 Source View, Target Rays of Different View

(1, none) 1 Source View, All-zero Rays

Source View: 2


(2, correct) 2 Source View, True Target Rays

(2, shuffled) 2 Source View, Target Rays of Different View

(2, none) 2 Source View, All-zero Rays

Source View: 4


(4, correct) 4 Source View, True Target Rays

(4, shuffled) 4 Source View, Target Rays of Different View

(4, none) 4 Source View, All-zero Rays

Metrics

Source Views Ray Type PSNR SSIM
1 Correct 15.11 0.55
1 Shuffled 11.84 0.49
1 None 12.46 0.53
2 Correct 14.90 0.55
2 Shuffled 11.94 0.49
2 None 12.41 0.53
4 Correct 16.64 0.56
4 Shuffled 12.16 0.51
4 None 12.43 0.54

Comparing the two models, training with 2 source views generally improves performance when correct ray conditioning is provided, particularly when evaluated with 2 views. However, the difference is much smaller when evaluated with 1 or 4 views. In contrast the model trained with 1 view performs slightly better when the rays are shuffled, although both models perform a lot worse under incorrect ray conditioning. The zero ray results are nearly identical, which means that the accuracy of the ray conditioning has a greater effect on reconstruction quality than the number of source views used during training.


Limitations

The difference between the models is relatively small in some settings, particularly at 1 and 4 evaluation views.
The experiment should be repeated with more views as well to see if there is a trend that is observed.


3. Multi-View Geometry with Mini-VGGT

3.1. Overfitting


Loss curve for the overfitting


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2


Ground Truth 3

Predicted 3

3.2. Training and Evaluation


Loss curve for the VGGT

Test Metrics

Metric Value
Depth AbsRel 0.1492
Depth δ1 0.8152

Test AUC Metrics

Metric AUC @ 5° AUC @ 15° AUC @ 30°
Rotation AUC 0.1671 0.4771 0.6929
Translation AUC 0.0184 0.0709 0.1753
Joint Pose AUC 0.0028 0.0378 0.1521

Validation Metrics

Metric Value
Depth AbsRel 0.1558
Depth δ1 0.8052

Validation AUC Metrics

Metric AUC @ 5° AUC @ 15° AUC @ 30°
Rotation AUC 0.1781 0.4679 0.6723
Translation AUC 0.0151 0.0720 0.1731
Joint Pose AUC 0.0028 0.0461 0.1473

Scene 1


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2

Scene 2


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2

Scene 3


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2

Scene 4


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2

Configuration used:

{ "data_config": "configs/scannetpp/iphone/data.json", "data_overrides": { "num_views": 8, "sampling_range": 32, "eval_num_frames": 32, "batch_size": 8, "num_workers": 4 }, "model": { "embed_dim": 384, "num_heads": 6, "num_layers": 8, "predict_points": false, "encoder_name": "dinov2_vitb14", "image_size": 224 }, "loss": { "camera": 5.0, "depth": 1.0 }, "training": { "epochs": 1000, "learning_rate": 0.0006, "weight_decay": 0.01, "output_dir": "outputs/vggt", "overfit_scenes": 1, "overfit_steps_per_epoch": 25, "min_lr_ratio": 0.05, "log_every": 10, "warmup_epochs": 10, "overfit_epochs": 40, "overfit_learning_rate": 0.001 } }

For held out scenes (validation and testing) the model recovered the geometry of the scene decently well, but made errors in smaller objects, camera alignment and certain boundaries for far away objects. Test and validation metrics are decent, with reasonable generalization, without overfitting. The model produces decent scenes but the predictions are still approximate in ambiguous regions.


3.3. Bonus: Analysis and Exploration

Unfreezing DINO: How does training the DINO encoder affect performance compared with keeping it frozen?

For this experiment, the DINO weights were made trainable, and the entire model was trained with the same configurations.


Loss curve for the VGGT with unfrozen DINO

Test Metrics

Metric Standard VGGT Unfrozen DINO
Depth AbsRel 0.1492 0.2848
Depth δ1 0.8152 0.5798

Test AUC Metrics

Metric AUC @ 5° (Standard / DINO) AUC @ 15° (Standard / DINO) AUC @ 30° (Standard / DINO)
Rotation AUC 0.1671 / 0.1079 0.4771 / 0.3174 0.6929 / 0.5354
Translation AUC 0.0184 / 0.0048 0.0709 / 0.0310 0.1753 / 0.1037
Joint Pose AUC 0.0028 / 0.0015 0.0378 / 0.0166 0.1521 / 0.0860

Validation Metrics

Metric Standard VGGT Unfrozen DINO
Depth AbsRel 0.1558 0.2479
Depth δ1 0.8052 0.6307

Validation AUC Metrics

Metric AUC @ 5° (Standard / DINO) AUC @ 15° (Standard / DINO) AUC @ 30° (Standard / DINO)
Rotation AUC 0.1781 / 0.0962 0.4679 / 0.2941 0.6723 / 0.4993
Translation AUC 0.0151 / 0.0015 0.0720 / 0.0291 0.1731 / 0.1014
Joint Pose AUC 0.0028 / 0.0000 0.0461 / 0.0149 0.1473 / 0.0818

Observations

Unfreezing the DINO backbone resulted in worse performances across every reported metric compared to the standard VGGT with a frozen DINO backbone. This suggests that fine-tuning the DINO backbone under the current setup negatively impacted the learned parameters. This is probably because the model was not trained for long enough and with enough data.


Scene 1


Ground Truth 1

Predicted 1


Ground Truth 2

Predicted 2

Scene 2


Ground Truth 1

Predicted 1