Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Both viewers below come from my own Q1 code. Use the menus to pick a scene and a view.

After unprojection: world frame, meters
Unprojection screenshot. Original world frame, in meters.
After canonicalization: camera 0 frame, normalized units
Canonicalization screenshot. Camera 0 center and valid-point centroid distance are in the side panel.

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Reprojection errors are around 1e-4 pixels, well under the 0.01 limit. The centroid distance is 1 for all four samples.

Explanations

Canonicalization. Every scene has its own world frame and its own size. That makes the data inconsistent across scenes. So I express all cameras relative to camera 0 (T_i' = T_0-1 T_i). Camera 0 becomes the identity. Then I divide translations, depth and points by one scale per scene. The scale is chosen so the centroid of the valid points is 1 unit from camera 0. Now every sample uses the same frame and scale.

World-space ray directions. For pixel (u, v) the direction is d = normalize(R K-1 [u, v, 1]T). K is the resized intrinsics and R is the camera-to-world rotation. It is a direction, so translation plays no part. The ray origin o is the camera center, the same for every pixel in a view.

Plücker moment. A ray is written as (d, m) with m = o × d. Slide the origin along the ray to o + s d and m stays the same, because (o + s d) × d = o × d. So (d, m) describes the line itself, not the point I picked as origin. For a unit d, the length of m is the distance from the line to the world origin.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

I trained on one fixed batch for 2000 steps. The final PSNR is 30.92 dB (SSIM 0.870). That meets the ~30 dB target. Below is val_epoch000.png from that run.

Overfit run: source views, ground truth, prediction

2.2 Training and Evaluation

Settings. I used configs/lvsm.json. Patch size 8, width 384, 8 layers, head dim 64, S=2 source views. AdamW with learning rate 0.0002 and weight decay 0.05. 200 warmup steps, then cosine decay. Gradient clipping at 1.0, bfloat16 autocast, batch size 4. I ran 100 epochs.

Training curves from log.jsonl

Validation PSNR goes from 12.25 dB at epoch 0 to 16.14 dB at epoch 99. The best checkpoint is epoch 99, and I used it for evaluation.

Validation PSNR / SSIMcorrect raysshuffled raysno rays
S = 115.19 / 0.558411.80 / 0.487712.45 / 0.5284
S = 216.14 / 0.560711.48 / 0.472612.42 / 0.5270
S = 416.88 / 0.575411.59 / 0.473312.44 / 0.5380

Number of source views. With correct rays, PSNR is 15.19 dB at S=1, 16.14 dB at S=2 and 16.88 dB at S=4. I only trained with S=2. PSNR still goes up with every extra view, even for S=1 and S=4. More views probably cover more of the target, so the model guesses less. SSIM goes from 0.558 to 0.575.

Ray conditioning. At S=2, correct rays give 16.14 dB. No rays gives 12.42 dB (3.72 lower). Shuffled rays gives 11.48 dB (4.66 lower). The target rays are the only input that says which view to render. Without them the model can only output something that does not depend on the view.

Shuffled rays are worse than no rays. With wrong rays the model commits to a sharp guess at the wrong viewpoint. MSE punishes that more than a blurry average.

Changing the rays moves PSNR by about 4.7 dB. Changing the number of source views moves it by about 1.7 dB. So the target pose matters much more than the number of sources.

Training visualizations (validation batch)

val_epoch000.png (source views, ground truth, prediction)
val_epoch050.png (source views, ground truth, prediction)
val_epoch095.png (source views, ground truth, prediction)

Evaluation visualizations (source views, ground truth, prediction)

S=1, correct rays
S=2, correct rays
S=4, correct rays
S=2, shuffled rays
S=2, no rays

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

I fit one fixed training clip with the default overfit settings (40 epochs of 25 steps). Total loss goes from 5.48 to -0.16. It ends below zero because the confidence term has -0.2 log(confidence), which is negative once confidence is above 1. On the clip, AbsRel is 0.036 and delta1 is 0.955. Rotation error is 0.23 deg and translation error is 0.46 deg. Pose AUC@30 is 0.995.

With one clip there is nothing to generalize to. This only shows the code can fit both cameras and depth. Held-out results are in 3.2.

Overfit loss curve (total, camera, depth)
Overfit scene viewer. Switch between GT and prediction in the second menu.

Overfit scene: GT vs prediction


Ground truth

Prediction

The predicted cameras sit close to the GT ones and the point cloud has the same shape. The predicted depth map is a bit blockier than the GT.

3.2 Training and Evaluation

Settings. I used configs/vggt.json. num_views is 8 and sampling_range is 32, unchanged. Batch size 16, learning rate 0.0006, weight decay 0.01, cosine schedule. The default is 1000 epochs. That would not fit in my time, so I trained for 200 epochs (14 steps per epoch, 2,800 steps). The schedule was set for 200 epochs, so the learning rate decays fully. The best checkpoint is chosen by validation loss.

Training loss curve (total, camera, depth)

Validation camera loss goes from 0.171 at epoch 101 to 0.163 at epoch 200. Validation depth loss goes from 0.351 to 0.362, so it is flat. The best validation loss is at epoch 190.

SplitAbsReldelta1Rot AUC 5/15/30Trans AUC 5/15/30Pose AUC 5/15/30Rot / trans err (deg)
val0.1770.7570.119 / 0.307 / 0.4770.007 / 0.046 / 0.1080.003 / 0.032 / 0.08824.9 / 50.9
test0.1880.7340.142 / 0.345 / 0.5360.004 / 0.038 / 0.0970.002 / 0.022 / 0.08019.7 / 52.9

28 scenes per split, 8 views each, best.pt. I evaluated the test split once.

Errors on held-out scenes

Depth is learned better than the cameras. AbsRel is 0.177 on val and 0.188 on test. Delta1 is 0.757 and 0.734. Val and test are close, so the model is not tuned to the val scenes.

The cameras are weak, translation most of all. Rotation AUC@30 is 0.477 on val and 0.536 on test. Translation AUC@30 is only 0.108 and 0.097. The pose AUC uses the worse error of the two for each pair, so translation sets it (0.088 val, 0.080 test). The mean errors say the same. Rotation is 24.9 / 19.7 deg and translation direction is 50.9 / 52.9 deg (val / test).

Test rotation is better than val, but test depth is slightly worse. With 28 scenes per split I would not read much into that. The bigger limit is training length. 200 epochs is a fifth of the default. Cameras are slower to learn, so I expect them to be well below what the full schedule gives.

What the viewer shows. In the four validation scenes the predicted cameras bunch up near camera 0. In 09c1414f1b and 2b1dc6d6a5 they collapse into one small cluster. The GT cameras are spread out. In 2b1dc6d6a5 the GT cameras sit in a row, and the predicted ones don't. So the model underestimates how far the cameras move. That matches the low translation AUC.

The room layout is right in all four scenes. Floor, walls and hallway are in the right place. Fine detail is lost. In 09c1414f1b the table and chairs blur into one blob. In 21d970d8de the predicted cloud is a smooth bowl, and the chair and table structure from the GT is mostly gone. In 280b83fcf3 the hallway comes out shorter and blobbier. The predicted depth maps are smooth. The GT ones have sharp edges.

The scale is roughly right but not exact. The valid-point centroid distance should be 1.0. For the predictions it is 0.926, 1.053, 0.981 and 1.094 for the four scenes below, in order. So the predicted points are off by up to about 9%.

Validation scenes viewer. Switch between GT and prediction in the second menu.

Validation scene 1 (09c1414f1b)


Ground truth

Prediction

Validation scene 2 (21d970d8de)


Ground truth

Prediction

Validation scene 3 (280b83fcf3)


Ground truth

Prediction

Validation scene 4 (2b1dc6d6a5)


Ground truth

Prediction

3.3 Bonus: number of views

Question. I trained with 8 views. What happens if I give the model 2, 4, 8 or 16 views at evaluation time? I changed only --num-views. Everything else matches the 3.2 evaluation. Same checkpoint, full val split.

VAbsReldelta1Rot AUC@30Trans AUC@30Pose AUC@30Pose pairs
20.1300.8360.4750.1150.08128
40.1610.7810.4540.1170.092168
80.1770.7570.4770.1080.088784
160.2170.6610.4390.1040.0843360

Results. Depth gets worse as I add views. AbsRel goes 0.130, 0.161, 0.177, 0.217 for V = 2, 4, 8, 16. Delta1 drops from 0.836 to 0.661. So the depth is best with fewer views than the 8 I trained with, not more.

The cameras show no clear trend. Pose AUC@30 stays between 0.081 and 0.092. Rotation AUC@30 is 0.475, 0.454, 0.477 and 0.439. I would not read much into those small differences.

Limitations. One checkpoint, one seed. The number of pose pairs changes a lot with V, from 28 to 3360. So the camera numbers are noisier at small V. All views come from the same first 32 frames. A bigger V probably adds near-duplicate frames and not much new content. I only measured accuracy, not runtime or memory.