16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Data visualizations

unprojection canonicalization

check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010844774078577757,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.281559323426336e-05,
      "max_ray_reprojection_error_px": 8.446346328128129e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010504279634915292,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization. Canonicalization is to put every views into the first view, so camera 0 becomes I. Other frames are projected into this.

World-space ray directions. For pixel (u, v) the camera-space direction is K^-1 [u, v, 1]^T. Rotating it by the camera-to-world rotation R and normalizing gives the world-space unit direction d.

Plücker moment. The moment is m = o * d. Any other point on the same ray gives the same moment, so (d, m) describes the ray itself rather than a particular point on it.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 30.83 dB (SSIM 0.870).

2.2 Training and Evaluation

Training settings:

Training curves

Validation PSNR / SSIM

source views S correct shuffled none
1 15.13 / 0.557 11.82 / 0.489 12.44 / 0.531
2 16.05 / 0.558 11.49 / 0.474 12.43 / 0.529
4 16.68 / 0.573 11.63 / 0.478 12.48 / 0.539

Training visualization (validation, epoch 95)

Columns: source views, ground truth, prediction.

Evaluation visualizations

Columns: source views, ground truth, prediction.

S correct shuffled none
1
2
4

Discussion

Number of source views. With the correct rays, PSNR goes from 15.13 (S = 1) to 16.05 (S = 2) to 16.68 dB (S = 4), even though the model was trained with S = 2 only. More source views cover more of the scene that the target camera sees, so less of the target has to be guessed.

Ray conditioning. Without target rays the model prediction starts to get blurry, roughly average-colored image for every target, about 12.4 dB. With another view's rays it renders a scene layout that belongs to a different viewpoint, so windows and furniture show up in the wrong places. This is even worse than none (about 11.5-11.8 dB), because putting structure in the wrong place costs more MSE than predicting a blurry average.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

loss curve (total, camera, depth)
input view 0 GT geometry predicted geometry

3.2 Training and Evaluation

My training settings:

Loss curves

Metrics

8 views from the first 32 prepared frames, 224px, seed 825.

split AbsRel ↓ δ1 ↑ rotation AUC @5/15/30 ↑ translation AUC @5/15/30 ↑ pose AUC @5/15/30 ↑
val 0.163 0.793 0.187 / 0.502 / 0.690 0.007 / 0.050 / 0.148 0.004 / 0.041 / 0.136
test 0.172 0.792 0.190 / 0.498 / 0.702 0.002 / 0.051 / 0.157 0.000 / 0.036 / 0.140

GT vs. predicted geometry (four validation scenes)

input view 0 GT geometry predicted geometry

Discussion

I visualized a few samples as if they were in the viewpoint of the camera. Comparing Ground Truth and the Prediction, I can see the points prediction seems to be blend toghther (unlike GT, it has a clear contour for example chairs).