Assignment 4: Multi-View Transformers

Name: Yannick Daniel Gibson
Andrew ID: ydg

1. Multi-View Data Preprocessing (20 points)

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 9.650505671743304e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.281559323426336e-05,
      "max_ray_reprojection_error_px": 8.446346328128129e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010796334390761331,
      "max_ray_reprojection_error_px": 9.860387945082039e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Training scene 1

Original world frame (meters)
Original world frame (meters)
First-camera frame (normalized)
First-camera frame (normalized)

Training scene 2

Original world frame (meters)
Original world frame (meters)
First-camera frame (normalized)
First-camera frame (normalized)

Canonicalization transforms cameras and points into the first camera frame and divides translations, depth and points by the valid-point centroid distance. World-space ray directions are normalized inverse-intrinsics pixel directions rotated by the camera pose. The Plucker moment is the camera center cross the ray direction.

2. Novel-View Synthesis with Mini-LVSM (40 points)

2.1. Fixed-batch overfitting (10 points)

After 2000 updates on one fixed batch, overfit PSNR reaches 38.62 dB.

Target 1

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

Target 2

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

Target 3

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

Target 4

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

2.2. Training and evaluation (30 points)

Training uses 8 views, 2 source views, 224-pixel images, 16-pixel patches, width 384, 8 transformer blocks, batch size 2 and AdamW with peak learning rate 0.0002 for 100 epochs.

Full training: training and validation MSE.
Full training: training and validation MSE.
Full training: training and validation PSNR (dB).
Full training: training and validation PSNR (dB).

Training visualization from epoch 90; the ablation images below use the best validation checkpoint.

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction
Validation settingPSNR (dB)SSIM
1 source view, correct target rays15.050.5569
1 source view, shuffled target rays11.750.4890
1 source view, no target rays12.480.5144
2 source views, correct target rays15.940.5564
2 source views, shuffled target rays11.470.4734
2 source views, no target rays12.460.5121
4 source views, correct target rays16.540.5691
4 source views, shuffled target rays11.570.4737
4 source views, no target rays12.470.5219

1 source view, correct target rays

Source view 1
Source view 1
Ground truth
Ground truth
Prediction
Prediction

1 source view, shuffled target rays

Source view 1
Source view 1
Ground truth
Ground truth
Prediction
Prediction

1 source view, no target rays

Source view 1
Source view 1
Ground truth
Ground truth
Prediction
Prediction

2 source views, correct target rays

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

2 source views, shuffled target rays

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

2 source views, no target rays

Source view 1
Source view 1
Source view 2
Source view 2
Ground truth
Ground truth
Prediction
Prediction

4 source views, correct target rays

Source view 1
Source view 1
Source view 2
Source view 2
Source view 3
Source view 3
Source view 4
Source view 4
Ground truth
Ground truth
Prediction
Prediction

4 source views, shuffled target rays

Source view 1
Source view 1
Source view 2
Source view 2
Source view 3
Source view 3
Source view 4
Source view 4
Ground truth
Ground truth
Prediction
Prediction

4 source views, no target rays

Source view 1
Source view 1
Source view 2
Source view 2
Source view 3
Source view 3
Source view 4
Source view 4
Ground truth
Ground truth
Prediction
Prediction

With correct rays, increasing sources from 1 to 2 to 4 raises PSNR from 15.05 to 15.94 to 16.54 dB, although the remaining target views also change.

With two sources, correct rays score 15.94 dB, shuffled rays 11.47 dB and zero rays 12.46 dB; the model uses target-ray information, but its predictions remain blurry with visible patch boundaries.

2.3. Patch size (10 bonus points)

I compare 16-pixel and 32-pixel patches using the same training split, seed, resolution, source count, transformer width/depth and 100 epochs.

Evaluation uses identical source images, target images and rays across 28 test scenes and frame-sampling seeds 825, 826 and 827.

ConditionPSNR (dB)SSIMMedian inference (ms)Live memory (MB)
16-pixel patches16.13550.540839.079.3
32-pixel patches15.79730.539610.799.7

For both bonus experiments, runtime and memory are medians over 150 forward passes at batch size 1; memory measures live allocation after inference, not peak usage.

Validation example

Ground truth
Ground truth
16-pixel patches
16-pixel patches
32-pixel patches
32-pixel patches

The 32-pixel model runs 3.65 times faster with slightly lower PSNR and higher live memory: larger patches reduce attention tokens but enlarge the projection layers. Both predictions lose furniture detail, and the 32-pixel result is smoother; these single-seed training runs do not establish an optimal patch size.

3. Multi-View Geometry with Mini-VGGT (40 points)

3.1. Single-clip overfitting (10 points)

Single-clip overfitting: total, camera and depth loss.
Single-clip overfitting: total, camera and depth loss.

I fit one fixed eight-view training clip for 40 epochs.

Training scene 1

Ground truth
Ground truth
Prediction
Prediction

3.2. Training and held-out evaluation (30 points)

Training and held-out evaluation: total, camera and depth loss.
Training and held-out evaluation: total, camera and depth loss.

Training uses frozen DINOv2 ViT-B/14, width 384, 8 frame/global block pairs, 8 views, sampling range 32, 224-pixel images, batch size 2, AdamW learning rate 0.0006 and 100 epochs; the best validation checkpoint is epoch 96.

Validation scene 1

Ground truth
Ground truth
Prediction
Prediction

Validation scene 2

Ground truth
Ground truth
Prediction
Prediction

Validation scene 3

Ground truth
Ground truth
Prediction
Prediction

Validation scene 4

Ground truth
Ground truth
Prediction
Prediction
MetricValidationTest
Depth AbsRel0.17840.1934
Depth delta10.76120.7352
Rotation AUC at 5 deg0.11990.1311
Rotation AUC at 15 deg0.27730.3086
Rotation AUC at 30 deg0.44100.4979
Translation AUC at 5 deg0.00410.0056
Translation AUC at 15 deg0.03940.0303
Translation AUC at 30 deg0.10040.0888
Joint pose AUC at 5 deg0.00260.0041
Joint pose AUC at 15 deg0.02870.0215
Joint pose AUC at 30 deg0.08410.0746

Evaluation uses both full splits, with 8 views from the first 32 prepared frames, 224-pixel images and seed 825.

Scene 1 merges separated surfaces, scene 2 makes surfaces fuzzy and clusters cameras, scene 3 obscures the hallway opening and introduces tall structures near its entrance that are absent from the ground truth, and scene 4 clusters camera positions and duplicates surfaces; the model recovers coarse extent but struggles to fuse held-out views.

3.3. Number of input views (10 bonus points)

I compare 2, 4 and 8 input views using one fixed checkpoint across 28 test scenes and frame-sampling seeds 825, 826 and 827, scoring the same first two views and camera pair with unchanged eight-view normalization.

ConditionDepth AbsRelDepth delta1Joint pose AUC at 30 degMedian inference (ms)Live memory (MB)
2 input views0.19420.73840.059947.6585.3
4 input views0.18960.73920.061584.2587.3
8 input views0.19320.73220.0690165.7592.0

Additional views provide no consistent improvement across the depth and pose metrics, while runtime and live memory increase because more views require more encoder and attention work; conclusions are limited to this checkpoint, which was trained with eight views.

Test example: same two scored views

Ground truth
Ground truth
2 input views
2 input views
4 input views
4 input views
8 input views
8 input views