16-825 Assignment 4: Multi-View Transformers

All training ran on one NVIDIA T4 GPU (15 GB).

1. Multi-View Data Preprocessing

After unprojection

Click an image for full size.

Scene 036bce3393Scene 079a326597
Unprojection, scene 036bce3393 Unprojection, scene 079a326597

After canonicalization

In both scenes the first camera center is (0.0000, 0.0000, 0.0000) and the valid-point centroid distance is 1.000. These values are also printed under each plot. Click an image for full size.

Scene 036bce3393Scene 079a326597
Canonicalization, scene 036bce3393 Canonicalization, scene 079a326597

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010844774078577757,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.281559323426336e-05,
      "max_ray_reprojection_error_px": 8.446346328128129e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010504279634915292,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. Left-multiplying every pose by the inverse of the first camera's pose places that camera at the origin with no rotation. Translations, depth and points are then divided by s = ||mean(p_valid)||, how far the average valid point lies from the origin. This gives every scene the same reference frame and scale.

World-space ray directions. For pixel (u, v), K⁻¹[u, v, 1]ᵀ is the camera-space direction. Applying the camera's rotation and dividing by the length gives d, the unit direction in canonical coordinates.

Plücker moment. m = o × d, where o is the camera position. Any point on the ray gives the same m, since (o + td) × d = o × d, so [d, m] describes the line itself. Without m, parallel rays from different cameras would look identical.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 33.82 dB (SSIM 0.9118), after 2,000 steps at batch size 2.

Overfit val_epoch000.png
val_epoch000.png. Columns: source views, ground truth, prediction.

2.2 Training and evaluation

Settings. Batch size 2 and 50 epochs (5,600 steps), 3.5 hours. The config default is batch size 4 and 100 epochs, but batch size 4 did not fit in GPU memory. All other settings are the defaults in configs/lvsm.json, including 2 source views and learning rate 2e-4. The best checkpoint is epoch 46.

Training curves
Training curves.

Validation PSNR (dB) / SSIM

Source viewsCorrect raysShuffled raysNo rays
114.94 / 0.553111.77 / 0.490312.45 / 0.5362
215.65 / 0.553211.54 / 0.479812.40 / 0.5322
416.14 / 0.566711.69 / 0.485812.42 / 0.5411

Training visualizations

Columns within each image: source views, ground truth, prediction. Click an image for full size.

Epoch 0Epoch 25Epoch 45
Epoch 0 Epoch 25 Epoch 45

Evaluation visualizations

Columns within each image: source views, ground truth, prediction. Click an image for full size.

Source viewsCorrect raysShuffled raysNo rays
1 1 source view, correct rays 1 source view, shuffled rays 1 source view, no rays
2 2 source views, correct rays 2 source views, shuffled rays 2 source views, no rays
4 4 source views, correct rays 4 source views, shuffled rays 4 source views, no rays

Discussion

Source views. Given the true rays, PSNR climbs with each added view, from 14.94 dB with one source to 16.14 dB with four, because more of each target is visible in the inputs.

Ray conditioning. The requested viewpoint reaches the model only through the target rays. With no rays, PSNR stays near 12.4 dB at every view count and the output is a flat average color. Shuffled rays score lowest (11.5–11.8 dB) because the model renders the wrong view, which costs more under MSE than an average.

In all settings the predictions are blurry. With correct rays they still get the coarse layout right.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit loss curves
Overfit loss curves. The total can go negative because of the confidence log term.

Click an image for full size.

Ground truthPrediction
Overfit scene, ground truth Overfit scene, prediction

3.2 Training and evaluation

Settings. Batch size 8 and 300 epochs (8,400 steps), 9.9 hours. The config default is batch size 16 and 1,000 epochs, but batch size 16 did not fit in GPU memory and I shortened the run to fit the time available. All other settings are the defaults in configs/vggt.json, including 8 views, sampling range 32 and learning rate 6e-4. Metrics use best.pt from epoch 297.

Normal-training loss curves
Normal-training loss curves.

Metrics

MetricValidationTest
Depth AbsRel0.17630.1878
Depth δ10.75880.7374
Rotation AUC @ 5°0.13900.1298
Rotation AUC @ 15°0.38600.4034
Rotation AUC @ 30°0.56990.6301
Translation AUC @ 5°0.00690.0056
Translation AUC @ 15°0.05110.0412
Translation AUC @ 30°0.12640.1232
Joint pose AUC @ 5°0.00260.0010
Joint pose AUC @ 15°0.03750.0291
Joint pose AUC @ 30°0.11000.1109

Ground truth vs prediction on four validation scenes

Click an image for full size.

SceneGround truthPrediction
1 Scene 1, ground truth Scene 1, prediction
2 Scene 2, ground truth Scene 2, prediction
3 Scene 3, ground truth Scene 3, prediction
4 Scene 4, ground truth Scene 4, prediction

Errors on held-out scenes

Depth. Walls, floors and large furniture are captured, but thin objects like chair legs are lost, and flat walls look thick and noisy in the point clouds.

Rotation. Rough orientation is mostly right (AUC 0.57–0.63 at 30°), but precise alignment is not (about 0.13–0.14 at 5°).

Translation. This is the weakest result, with a mean direction error of 49.2° on validation and 46.8° on test, and it limits joint pose AUC. Predicted cameras bunch near the first camera instead of following the true path, most clearly in scene 4. A likely cause is that small translations are a low-loss choice when the model is uncertain.