16-825 Assignment 4: Multi-View Transformers

Name: Hiroki Mii

Andrew ID: hmii

1. Multi-View Data Preprocessing

1.2 Depth Unprojection

Scene 1 Scene 2 Scene 3 Scene 4 Scene 5

1.3 Canonicalization

Scene 1
Scene 2
Scene 3
Scene 4
Scene 5

1.7 Final Check


{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Canonicalization is expressing each sample in its first camera's coordinate frame and normalizing its scale. centroid_distance = 1.0 in every sample confirming the shared scale.

World-space ray directions is the unit direction of each pixel's ray in world-coordinates. It is computed through K-1[u, v, 1] (the direction in the camera's own axes), rotated into the world frame by the camera's rotation matrix, and normalized to unit length.

Plücker moment is the cross product of a ray's origin and unit direction. A ray is represented in Plücker coordinates as [d, m], direction followed by moment.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

PSNR: 30.909

2.2 Training and Evaluation

Training Curves

Training Progression

val_epoch000 val_epoch045 val_epoch095

Ablations

S=1, conditioning=correct

PSNR = 15.193, SSIM = 0.558

S=1, conditioning=shuffled

PSNR = 11.797, SSIM = 0.487

S=1, conditioning=none

PSNR = 12.468, SSIM = 0.528

S=2, conditioning=correct

PSNR = 16.124, SSIM = 0.560

S=2, conditioning=shuffled

PSNR = 11.485, SSIM = 0.472

S=2, conditioning=none

PSNR = 12.448, SSIM = 0.527

S=4, conditioning=correct

PSNR = 16.876, SSIM = 0.575

S=4, conditioning=shuffled

PSNR = 11.577, SSIM = 0.472

S=4, conditioning=none

PSNR = 12.460, SSIM = 0.538

The training configurations were left as is in configs/lvsm.json with epochs=100, batch_size=4, learning_rate=0.0002, patch_size=8, d_model=384, n_layers=8. The results of the ablations were consistent to what was expected. As the number of source views increased, both PSNR and SSIM scores increased (with correct rays). More source views cover more of the target view, so the model has more of the scene to draw from. For ray conditioning, correct uses true target rays so it got the highest PSNR and SSIM scores. Shuffled and None both got lower scores because the model no longer knows which view to render. Shuffled gives it rays of a different view, and None gives it no camera information.

2.3 Bonus: Loss Functions

We investigate how adding an L1 or SSIM term to the MSE loss changes PSNR, SSIM, and visual sharpness. All other training configurations were left as is from 2.2.

MSE + L1

Training Curves S=1, conditioning=correct

PSNR = 15.328, SSIM = 0.559

S=2, conditioning=correct

PSNR = 16.287, SSIM = 0.563

S=4, conditioning=correct

PSNR = 17.096, SSIM = 0.579

MSE + SSIM

Training Curves S=1, conditioning=correct

PSNR = 14.876, SSIM = 0.575

S=2, conditioning=correct

PSNR = 15.551, SSIM = 0.578

S=4, conditioning=correct

PSNR = 15.975, SSIM = 0.591

Neither the addition of an L1 or SSIM term contributed to significant improvements. L1 term raised PSNR by about 0.1-0.2 dB with no change in SSIM, while SSIM term raised SSIM by about 0.02, but lowered PSNR by about 0.3-0.9. Like 2.2, the predicted outputs are still blocky, and the continued decrease in training and validation loss indicates the model is underfitted and needs further training. The loss functions can be re-evaluated after training for more epochs.

Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Loss Curves

Ground Truth (left) vs Prediction (right)

3.2 Training and Evaluation

Loss Curves

Validation Metrics


  "depth_abs_rel": 0.13722808236251358,
  "depth_delta1": 0.8515819910086473,
  "rotation_auc_3": 0.11224489795918367,
  "rotation_auc_5": 0.2114795918367347,
  "rotation_auc_15": 0.5318877551020408,
  "rotation_auc_30": 0.7153911564625851,
  "translation_auc_3": 0.004251700680272109,
  "translation_auc_5": 0.012244897959183673,
  "translation_auc_15": 0.08095238095238094,
  "translation_auc_30": 0.2037840136054422,
  "pose_auc_3": 0.0,
  "pose_auc_5": 0.004081632653061224,
  "pose_auc_15": 0.06301020408163266,
  "pose_auc_30": 0.1876700680272109

Test Metrics


  "depth_abs_rel": 0.15694556708130455,
  "depth_delta1": 0.7960929582679249,
  "rotation_auc_3": 0.13477891156462585,
  "rotation_auc_5": 0.2298469387755102,
  "rotation_auc_15": 0.5437074829931973,
  "rotation_auc_30": 0.734141156462585,
  "translation_auc_3": 0.005527210884353741,
  "translation_auc_5": 0.013520408163265307,
  "translation_auc_15": 0.08112244897959184,
  "translation_auc_30": 0.1993197278911565,
  "pose_auc_3": 0.0012755102040816326,
  "pose_auc_5": 0.004591836734693878,
  "pose_auc_15": 0.05748299319727891,
  "pose_auc_30": 0.18142006802721092

Visualizations

Ground truths on left, predictions on right.

Scene 1
Scene 2
Scene 3
Scene 4

Besides changing to num_workers=4 to avoid OOM errors, training configurations were left as is in configs/vggt.json with epochs=1000, batch_size=16, learning_rate=0.0006, num_views=8, sampling_range=32. On held-out scenes, depth and rotation are reasonably accurate. depth_abs_rel was 0.137 on validation, and 0.157 on test. rotation_auc_30 was 0.715 on validation, and 0.734 on test. Translation was the weakest with translation_auc_30 0.204 on validation, and 0.199 on test. Training and validation loss were still decreasing at epoch 1000 so longer training could help. In the visualizations, the predicted point clouds capture the room layout and furniture, but surfaces are still a little noisy with stray points and smearing.