Multi-View Transformers

henrytsu · 16-825 · Assignment 4

1. Multi-View Data Preprocessing

1.1–1.2 Intrinsics and depth unprojection

After center cropping, intrinsics follow the half-pixel resize convention: f′ = s f, c′ = s(c + 0.5) − 0.5. For each pixel p = [u, v, 1], optical-axis depth gives X_camera = depth · K⁻¹p; the camera-to-world pose maps it to X_world = R X_camera + t.

Unprojected training points and cameras in meters

1.3 Canonicalization

All poses are expressed relative to the first sampled camera. Translations, depths and points are divided by ||mean(valid points)||, without subtracting the centroid. This fixes camera 0 at the origin and puts the valid-point centroid one unit away, giving samples a shared coordinate and scale convention.

Canonicalized points with first camera center and centroid distance

The screenshot reports camera 0 at (0, 0, 0) and centroid distance 1.000.

1.4–1.6 Rays, Plücker coordinates and sampling

A ray starts at the camera center. Its world direction is normalize(R K⁻¹p). The six Plücker channels are [d, o × d]: direction and moment, which jointly represent the oriented line. Training samples eight randomly ordered frames within a 32-frame window; the first sampled view sets the reference frame. Prepared validation/test samples use fixed views.

1.7 Data check

Four training samples passed the geometry check. Each uses eight 224 × 224 views; all centroid distances are 1.000.

Sample Point reprojection error (px) Ray reprojection error (px) Valid depth
e050c15a8d_frame_007080 0.000111 0.000108 97.79%
5f99900f09_frame_002820 0.000123 0.000111 99.03%
47b37eb6f9_frame_002720 0.000102 0.000089 96.10%
4c5c60fa76_frame_006690 0.000109 0.000103 75.17%
Full check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011141905270051211,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.0 Implementation and settings

Source tokens contain RGB and Plücker rays; target tokens contain only rays. Each target attends to the source views independently. The transformer decodes target tokens into RGB patches and learns with pixel MSE.

Setting Value
Train / validation / test scenes 224 / 28 / 28
Views / image size 8 / 224 × 224
Source views during training 2
Patch size / width / blocks / heads 8 / 384 / 8 / 6
Batch size / epochs 4 / 100
Optimizer AdamW, lr 0.0002, weight decay 0.05, betas (0.9, 0.95)
Schedule 200-step warmup, cosine decay to 10% of peak lr
Precision / gradient clipping / seed bfloat16 / 1.0 / 825
Hardware NVIDIA RTX 5090, 32 GB

2.1 Overfitting

After 2,000 steps on one fixed batch: PSNR 30.91 dB, SSIM 0.8697.

LVSM overfit: source views, ground truth, prediction

Columns: two source views, target ground truth, prediction.

2.2 Training and evaluation

LVSM training MSE and PSNR

Validation visualization during training, epoch 96

Training visualization at epoch 96; columns follow the same source / GT / prediction order. Evaluation below uses the best validation checkpoint, epoch 100, on all 28 validation scenes. Views are selected from the first 32 prepared frames.

Sources Target-ray conditioning PSNR (dB) SSIM Target images
1 correct 15.21 0.5585 196
1 shuffled 11.81 0.4877 196
1 none 12.46 0.5292 196
2 correct 16.12 0.5601 168
2 shuffled 11.49 0.4724 168
2 none 12.43 0.5267 168
4 correct 16.85 0.5751 112
4 shuffled 11.58 0.4734 112
4 none 12.45 0.5362 112

With correct rays, PSNR rises from 15.21 to 16.85 dB as the source count increases from one to four. This comparison also changes the number and identity of target views. Shuffling target rays drops PSNR below 12 dB; removing them gives about 12.4 dB. The model uses camera conditioning, but held-out predictions remain blurry and less detailed than the overfit result.

1 source view

Correct target rays with 1 source views

Shuffled and missing target rays

Shuffled target rays

Missing target rays

2 source views

Correct target rays with 2 source views

Shuffled and missing target rays

Shuffled target rays

Missing target rays

4 source views

Correct target rays with 4 source views

Shuffled and missing target rays

Shuffled target rays

Missing target rays

Each row shows all source views, followed by target GT and prediction. Download LVSM metrics.

3. Multi-View Geometry with Mini-VGGT

3.0 Implementation and settings

A frozen DINOv2 ViT-B/14 encoder provides image features. Eight pairs of frame/global attention blocks exchange information within and across views. Concatenated final-pair features feed camera and depth/confidence heads. Camera supervision uses translation, quaternion and FoV errors; depth supervision combines trimmed regression, confidence weighting and four-scale residual gradients.

Setting Value
Views / sampling range / image size 8 / 32 / 224 × 224
Width / attention heads / frame-global pairs 384 / 6 / 8
Batch size / epochs 16 / 1,000
Optimizer AdamW, lr 0.0006, weight decay 0.01
Schedule 10-epoch warmup, cosine decay to 5% of peak lr
Camera / depth loss weights 5 / 1
Precision / gradient clipping / seed bfloat16 / 1.0 / 825

3.1 Overfitting

One fixed eight-view training clip, 40 epochs × 25 steps, learning rate 0.001.

Depth AbsRel reaches 0.0376, δ1 0.9543, and mean rotation error 0.26°. The confidence term includes −0.2 log(confidence), so the combined depth loss can become negative.

VGGT overfit loss curves

Ground truth Prediction
Overfit ground-truth geometry Overfit predicted geometry

3.2 Training and evaluation

VGGT normal training and validation losses

The best checkpoint is epoch 929, selected by validation loss. Both complete splits use eight views from the first 32 prepared frames, 224 px, seed 825; no batch limit. Each split has 28 scenes and 784 camera pairs. AUC values below are on a 0–1 scale.

Metric Validation Test
Depth AbsRel ↓ 0.1433 0.1577
Depth δ1 ↑ 0.8337 0.7986
Rotation AUC @ 5° ↑ 0.1829 0.1765
Rotation AUC @ 15° ↑ 0.5055 0.5042
Rotation AUC @ 30° ↑ 0.6951 0.7159
Translation AUC @ 5° ↑ 0.0099 0.0128
Translation AUC @ 15° ↑ 0.0707 0.0495
Translation AUC @ 30° ↑ 0.1853 0.1440
Joint pose AUC @ 5° ↑ 0.0041 0.0015
Joint pose AUC @ 15° ↑ 0.0574 0.0280
Joint pose AUC @ 30° ↑ 0.1654 0.1281

The held-out depth errors are larger than the overfit error. Rotation is more reliable than translation direction: validation AUC@30° is 0.6951 for rotation versus 0.1853 for translation. Translation limits joint-pose accuracy and can misalign clouds from different views. Test depth AbsRel rises to 0.1577; low joint-pose AUC shows that good local depth alone does not guarantee a consistent multi-view reconstruction.

Four validation scenes are shown below with matching GT/prediction camera viewpoints and axis ranges.

Validation scene 1

Ground truth Prediction
Scene 1 GT Scene 1 prediction

Validation scene 2

Ground truth Prediction
Scene 2 GT Scene 2 prediction

Validation scene 3

Ground truth Prediction
Scene 3 GT Scene 3 prediction

Validation scene 4

Ground truth Prediction
Scene 4 GT Scene 4 prediction

Validation metrics · Test metrics.