Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Unprojection visualization:

Front view
Front view
Side view
Side view
Top-down view
Top-down view

Canonicalization visualization:

Front view
Front view
Side view
Side view
Top-down view
Top-down view

First camera center (X, Y, Z): (0.0000, -0.0000, -0.0000) normalized units.
Scene scale: 1.867 m.
Valid-point centroid distance: 1.000 normalized units.

check_data.py output (expand for json output):

Details ``` json { "split": "train", "dataset_samples": 224, "checked_samples": 4, "samples": [ { "sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.0001102154201362282, "max_ray_reprojection_error_px": 0.00010904046939685941, "centroid_distance": 1.0 }, { "sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00011108566104667261, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552 }, { "sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.5367431640625e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0 }, { "sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00010647465387592092, "max_ray_reprojection_error_px": 0.00010257320536766201, "centroid_distance": 0.9999999403953552 } ], "batch_shapes": { "rgb": [ 2, 8, 3, 224, 224 ], "depth": [ 2, 8, 1, 224, 224 ], "depth_valid": [ 2, 8, 1, 224, 224 ], "K": [ 2, 8, 3, 3 ], "frame_numbers": [ 2, 8 ], "T_world_from_camera": [ 2, 8, 4, 4 ], "original_world_from_reference": [ 2, 4, 4 ], "scene_scale": [ 2 ], "points_world": [ 2, 8, 3, 224, 224 ], "ray_origins": [ 2, 8, 3, 224, 224 ], "ray_directions": [ 2, 8, 3, 224, 224 ], "plucker": [ 2, 8, 6, 224, 224 ] } } ```

Canonicalization: The process of aligning 3D object data and camera poses into a default standardized format, adjusting scale, translation and pose. In this case, the scale and translation of the scene is adjusted such that the valid-point centroid lies at origin and the camera is a distance of 1 unit from the origin.

World-space ray directions: A 3D unit vector that describes the orientation of a camera ray with respect to the world coordinate system as opposed to the local camera coordinate system.

Plucker moment: This represents a continuous line in 3D space without specifying a starting point. The plucker moment describes the cross product of a point on the line with the direction vector of the line.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfitting PSNR:
training: 29.057 dB
validation: 36.616 dB

Visualization

visualization

2.2 Training and Evaluation

Training loss:

loss

PSNR ablation:

Ray-conditioning: Correct Shuffled None
1 source view 15.204 11.752 12.445
2 source views 16.121 11.435 12.428
4 source views 16.869 11.533 12.474

SSIM ablation:

Ray-conditioning: Correct Shuffled None
1 source view 0.554 0.483 0.525
2 source views 0.556 0.468 0.522
4 source views 0.572 0.469 0.533

Evaluation visualization with ablations:

Ray-conditioning: Correct Shuffled None
1 source view 1 correct 1 shuffled 1 none
2 source view 2 correct 2 shuffled 2 none
4 source view 4 correct 4 shuffled 4 none

Training visualization gif:

training visualization gif

Training settings (expand dropdown for json config):

batch_size was changed to 4 to fit into VRAM.

Details
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "batch_size": 4,
    "eval_num_frames": 32
  },
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": {},
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true,
    "vis_every": 5,
    "output_dir": "outputs/lvsm"
  },
  "num_source_views": 2
}

Increasing the number of source views from 1 to 2 yields the highest improvement in output while increasing it from 2 to 4 shows diminishing returns.

Changing the ray conditioning from correct to shuffled reduces the quality of the output drastically, which indicates that the model is using the Plucker rays as part of its attention mechanism. However changing it to none removes all directional context, rendering a blurry average output that is slightly better metrically, indicating a learned prior.

2.3 Analysis and exploration

Question: How does changing the training view count from 2 to 4 affect results and quality? How does this change the evaluation with 1, 2 and 4 source views?

PSNR ablation:

Ray-conditioning: Correct Shuffled None
1 source view 15.184 11.693 12.379
2 source views 16.297 11.329 12.346
4 source views 17.280 11.362 12.383

SSIM ablation:

Ray-conditioning: Correct Shuffled None
1 source view 0.552 0.477 0.523
2 source views 0.556 0.460 0.520
4 source views 0.573 0.456 0.529

Training visualization gif:

training visualization gif

Evaluation visualization with ablations:

Ray-conditioning: Correct Shuffled None
1 source view 1 correct 1 shuffled 1 none
2 source view 2 correct 2 shuffled 2 none
4 source view 4 correct 4 shuffled 4 none

Training settings (expand dropdown for json config):

num_source_views was changed from 2 to 4.

Details
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "batch_size": 4,
    "eval_num_frames": 32
  },
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": {},
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true,
    "vis_every": 5,
    "output_dir": "outputs/lvsm"
  },
  "num_source_views":4
}

Conclusion from experiment

Visually, compared to using 2 source views, the outputs from the model trained with 4 views produces visually slightly better outputs however the difference is slight, as demonstrated by the PSNR and SSIM score improvements as well.

This improvement seems to produce diminishing returns possibly because the jump from 1 to 2 contributes to parallax and 3D point triangulation while the jump from 2 to 4 adds mostly overlapping visual data.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Loss curves:

loss

GT vs Prediction

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

3.2. Training and Evaluation

Loss curves:

loss

GT vs Prediction: Scene 1

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

GT vs Prediction: Scene 2

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

GT vs Prediction: Scene 3

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

GT vs Prediction: Scene 4

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

Validation metrics

Metric Value
Depth AbsRel 0.146
Depth δ1 0.836
Rotation AUC 5 deg 0.168
Rotation AUC 15 deg 0.472
Rotation AUC 30 deg 0.657
Translation AUC 5 deg 0.009
Translation AUC 15 deg 0.066
Translation AUC 30 deg 0.170
Joint Pose AUC 5 deg 0.003
Joint Pose AUC 15 deg 0.051
Joint Pose AUC 30 deg 0.146

Test metrics

Metric Value
Depth AbsRel 0.162
Depth δ1 0.804
Rotation AUC 5 deg 0.182
Rotation AUC 15 deg 0.496
Rotation AUC 30 deg 0.691
Translation AUC 5 deg 0.012
Translation AUC 15 deg 0.052
Translation AUC 30 deg 0.143
Joint Pose AUC 5 deg 0.003
Joint Pose AUC 15 deg 0.039
Joint Pose AUC 30 deg 0.132

Training settings (expand dropdown for json config):

batch_size changed to 10 and num_workers set to 10 to fit in VRAM:

Details
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "num_views": 8,
    "sampling_range": 32,
    "eval_num_frames": 32,
    "batch_size": 10,
    "num_workers": 4
  },
  "model": {
    "embed_dim": 384,
    "num_heads": 6,
    "num_layers": 8,
    "predict_points": false,
    "encoder_name": "dinov2_vitb14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 1000,
    "learning_rate": 0.0006,
    "weight_decay": 0.01,
    "output_dir": "outputs/vggt",
    "overfit_scenes": 1,
    "overfit_steps_per_epoch": 25,
    "min_lr_ratio": 0.05,
    "log_every": 10,
    "warmup_epochs": 10,
    "overfit_epochs": 40,
    "overfit_learning_rate": 0.001
  }
}

The depth values of the points for held-out scenes are inaccurate possibly due to incorrect scale outputs.

3.3: Analysis and Exploration

Question: How does training the DINO encoder affect performance compared with keeping it frozen?

Experimental settings: Same as above but DINO gradients are enabled during training instead of frozen.

Loss curves:

loss

GT vs Prediction: Scene 1

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

GT vs Prediction: Scene 2

Front view (GT) Front view (Prediction)
Front view (GT) Front view (Prediction)
Side view (GT) Side view (Prediction)
Side view (GT) Side view (Prediction)
Top-down view (GT) Top-down view (Prediction)
Top-down view (GT) Top-down view (Prediction)

Validation metrics

Metric Value with unfrozen DINO backbone Baseline value Better
Depth AbsRel 0.237 0.146 Lower
Depth δ1 0.658 0.836 Higher
Rotation AUC 5 deg 0.115 0.168 Higher
Rotation AUC 15 deg 0.353 0.472 Higher
Rotation AUC 30 deg 0.552 0.657 Higher
Translation AUC 5 deg 0.002 0.009 Higher
Translation AUC 15 deg 0.025 0.066 Higher
Translation AUC 30 deg 0.092 0.170 Higher
Joint Pose AUC 5 deg 0.0005 0.003 Higher
Joint Pose AUC 15 deg 0.014 0.051 Higher
Joint Pose AUC 30 deg 0.071 0.146 Higher

Test metrics

Metric Value with unfrozen DINO backbone Baseline value Better
Depth AbsRel 0.273 0.162 Lower
Depth δ1 0.604 0.804 Higher
Rotation AUC 5 deg 0.140 0.182 Higher
Rotation AUC 15 deg 0.376 0.496 Higher
Rotation AUC 30 deg 0.577 0.691 Higher
Translation AUC 5 deg 0.003 0.012 Higher
Translation AUC 15 deg 0.042 0.052 Higher
Translation AUC 30 deg 0.124 0.143 Higher
Joint Pose AUC 5 deg 0.001 0.003 Higher
Joint Pose AUC 15 deg 0.039 0.039 Higher
Joint Pose AUC 30 deg 0.132 0.132 Higher

Conclusion from experiment

Visually, unfreezing the DINO backbone produces worse results, with depth errors more obviously seen.

This degradation is because the half the model is trained (the DINO backbone) while the other half (the VGGT heads) are untrained. This causes gradients to affect the trained backbone, disturbing the trained gradients. This effect is further amplified by the fact that both the frozen and unfrozen models are trained for the same number of training steps; the frozen model has way fewer parameters to optimize compared to the unfrozen model.