16-825 Assignment 4: Multi-View Transformers

Part 1. Multi-View Data Processing

1.1–1.2 Unprojection

Unprojection (original world frame, meters), scene 036bce3393
Unprojection, scene 076c822ecc

1.3 Canonicalization

Canonicalization, scene 036bce3393: first camera at (0, 0, 0), scene scale 1.867 m, centroid distance 1.000
Canonicalization, scene 076c822ecc: first camera at (0, 0, 0), scene scale 2.830 m, centroid distance 1.000

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ]
}
batch_shapes:
  rgb                            [2, 8, 3, 224, 224]
  depth                          [2, 8, 1, 224, 224]
  depth_valid                    [2, 8, 1, 224, 224]
  K                              [2, 8, 3, 3]
  frame_numbers                  [2, 8]
  T_world_from_camera            [2, 8, 4, 4]
  original_world_from_reference  [2, 4, 4]
  scene_scale                    [2]
  points_world                   [2, 8, 3, 224, 224]
  ray_origins                    [2, 8, 3, 224, 224]
  ray_directions                 [2, 8, 3, 224, 224]
  plucker                        [2, 8, 6, 224, 224]

All reprojection errors are about 1e-4 px (the limit is 0.01 px) and the valid-point centroid distance is 1.0 for every sample.

Explanations

Canonicalization.

World-space ray directions.

Plücker moment.

Part 2. Mini-LVSM

2.1 Overfitting

val_epoch000.png of the overfit run (columns: source views, ground truth, prediction)

2.2 Training and evaluation

Training settings. One A100, 100 epochs (5,600 steps), usually 24 s per epoch and 44 min in total. The best checkpoint is from epoch 99.

Full training config (configs/lvsm.json, resolved)
{
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": "MSE on target RGB",
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true
  },
  "num_source_views": 2,
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/scene_splits.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 4,
    "num_workers": 4,
    "seed": 825
  }
}
Training curves (loss, PSNR, SSIM). S=2 is the baseline; S=4 is the bonus run in 2.3.

Validation PSNR / SSIM for each ablation setting. All rows use the same model (trained with S = 2); only the number of source views given at test time and the target-ray conditioning change.

Source views at test timeCorrect raysShuffled raysNo rays
115.19 / 0.55811.80 / 0.48812.47 / 0.527
2 (trained)16.13 / 0.56011.49 / 0.47212.44 / 0.526
416.88 / 0.57511.59 / 0.47312.45 / 0.537

Training visualizations (columns: source views, ground truth, prediction):

Epoch 0
Epoch 95

Evaluation visualizations, ray-conditioning ablation (S = 2):

Correct target rays
Shuffled target rays
No target rays

Evaluation visualizations, number of source views (correct rays). One model, trained with S = 2, is given 1, 2 or 4 source views at test time:

Evaluated with S = 1
Evaluated with S = 2 (same as training)
Evaluated with S = 4
All other evaluation images
S = 1, shuffled
S = 1, none
S = 4, shuffled
S = 4, none

Number of source views.

Ray conditioning.

Quality.

2.3 Bonus: training view count

Question. If the model sees more source views during training (S = 4 instead of 2), how does this change evaluation with 1, 2 and 4 source views?

Settings. The same as the baseline, except 4 source views (so 4 targets per clip). One A100, about 27 s per epoch, 45 min. The best checkpoint is from epoch 95. Both models are evaluated with the same script, scenes and seed.

Full training config (configs/lvsm_s4.json, resolved)
{
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": "MSE on target RGB",
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true
  },
  "num_source_views": 4,
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/scene_splits.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 4,
    "num_workers": 4,
    "seed": 825
  }
}
Eval SConditioningTrained S=2: PSNR / SSIMTrained S=4: PSNR / SSIMΔ PSNR
1correct15.19 / 0.55815.33 / 0.557+0.14
2correct16.13 / 0.56016.41 / 0.562+0.27
4correct16.88 / 0.57517.46 / 0.580+0.58
1shuffled11.80 / 0.48811.75 / 0.478−0.06
2shuffled11.49 / 0.47211.36 / 0.460−0.13
4shuffled11.59 / 0.47311.40 / 0.457−0.19
1none12.47 / 0.52712.36 / 0.516−0.11
2none12.44 / 0.52612.33 / 0.512−0.12
4none12.45 / 0.53712.38 / 0.520−0.06
Trained S=4, evaluated S=1
Trained S=4, evaluated S=4

Results and conclusions.

Limitations.

Part 3. Mini-VGGT

3.1 Overfitting

Overfit loss curve (total, camera and depth loss)
Overfit clip, ground truth
Overfit clip, predicted camera + depth. The predicted geometry is almost the same as GT.

3.2 Training and evaluation

Training settings. Default config, two A100s with DDP (global batch 16), 1000 epochs, about 8 s per epoch for train + val and about 3.1 h wall-clock including checkpointing. The best validation loss is at epoch 950.

Full training config (configs/vggt.json, resolved)
{
  "model": {
    "embed_dim": 384,
    "num_heads": 6,
    "num_layers": 8,
    "predict_points": false,
    "encoder_name": "dinov2_vitb14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 1000,
    "learning_rate": 0.0006,
    "weight_decay": 0.01,
    "min_lr_ratio": 0.05,
    "warmup_epochs": 10
  },
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/scene_splits.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 16,
    "num_workers": 8,
    "seed": 825
  },
  "run": {
    "world_size": 2,
    "precision": "bf16"
  }
}
Training loss curve (total, camera and depth loss; train and validation)

Evaluation (best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825):

SplitAbs-rel ↓RMSE ↓δ1 ↑Rot. err ↓Trans. err ↓Rot. AUC @5/15/30Trans. AUC @5/15/30Pose AUC @5/15/30
Val0.1420.1900.83412.75°41.98°0.200 / 0.508 / 0.6900.013 / 0.075 / 0.1780.003 / 0.056 / 0.155
Test0.1550.1990.8019.51°40.53°0.191 / 0.535 / 0.7230.009 / 0.074 / 0.1840.001 / 0.057 / 0.166

GT vs. predicted geometry on four held-out validation scenes:

Val scene 09c1414f1b, ground truth
Val scene 09c1414f1b, predicted camera + depth
Val scene 21d970d8de, ground truth
Val scene 21d970d8de, predicted camera + depth
Val scene 280b83fcf3, ground truth
Val scene 280b83fcf3, predicted camera + depth
Val scene 2b1dc6d6a5, ground truth
Val scene 2b1dc6d6a5, predicted camera + depth

Errors on held-out scenes.

3.3 Bonus: number of input views

Question. How does the number of input views V affect accuracy, runtime and memory for a model trained with V = 8?

Settings.

VPose AUC@30Rot. AUC@30Trans. AUC@30Rot. errTrans. errAbs-relδ1Forward (ms)Peak memory (MB)
20.1960.7270.20611.24°33.66°0.1120.87764.1708
40.1500.6810.16812.04°41.24°0.1220.86656.6725
8 (trained)0.1550.6900.17912.75°41.96°0.1420.83476.7770
160.1390.6840.15813.73°44.27°0.1750.79590.7848

The V = 8 row differs from the Q3.2 val row in the last digit (translation error 41.96° vs 41.98°) because of bf16 nondeterminism and batch size 1 instead of 16.

Accuracy, forward time and peak GPU memory vs. number of input views (dotted line: training V)

Results and conclusions.