Assignment 4

1. Multi-View Data Preprocessing

1.1 - 1.2 Unprojection



1.3 Canonicalization



1.7 Final check

sequence                  valid depth   max reprojection (px)   max ray reprojection (px)   centroid distance
e050c15a8d_frame_007080   0.9779        1.13e-04                1.09e-04                    1.0
5f99900f09_frame_002820   0.9903        1.23e-04                1.11e-04                    1.0
47b37eb6f9_frame_002720   0.9610        9.78e-05                8.90e-05                    1.0
4c5c60fa76_frame_006690   0.7517        1.17e-04                1.07e-04                    1.0
  1. Canonicalization: Each sample (image) is taken by the camera in a different position, and so has a different world definition and scale, even though the world and scale of the underlying scene doesn't change. Canonicalization is then the process that involves moving all images to some fixed camera's (here, camera 0) world definition.
  2. World-space ray directions: For each pixel in an image, there is a ray originating from that image's camera and passing through that pixel in the image plane. These rays are transformed from that specific camera to camera 0.
  3. Plücker moment: m = o × d. Since (o + s d) × d = o × d, the moment is the same for any point o on the line and so we can use (d, m) to describe the line.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

PSNR = 30.91 dB

2.2 Training and Evaluation

"data_overrides": {
    "batch_size": 4,
    "eval_num_frames": 32
  },
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": {},
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true,
    "vis_every": 5,
    "output_dir": "outputs/lvsm"
  },
  "num_source_views": 2

Source viewsConditioningPSNRSSIM
1correct15.190.558
1shuffled11.810.488
1none12.470.528
2correct16.130.560
2shuffled11.500.473
2none12.450.527
4correct16.860.575
4shuffled11.600.473
4none12.480.537

Training visualization, epoch 95. Each row: sources | ground truth | prediction.

Correct rays. Left: 1 source, middle: 2 sources, right: 4 sources.

2 sources. Left: correct, middle: shuffled, right: none.

More sources views seem to help, leading to an increase in PSNR. More sources cover more of the scene, so each target pixel is more likely to be seen by at least one of them.

Without the correct rays, the PSNR drops. Also, it seems that having no rays at all is better than having the wrong rays (at least, for PSNR). With the wrong rays, the model may be predicting the right pixel in the wrong location, leading to a worse result for PSNR (but better, visually).

2.3 Bonus: SSIM loss

Since the results look blurry either way, I experimented with adding a penalty for lower SSIM (weight of 0.1) since that also computes local statistics.

Loss1 source PSNR / SSIM2 sources PSNR / SSIM4 sources PSNR / SSIM
MSE15.19 / 0.55816.13 / 0.56016.86 / 0.575
MSE + SSIM14.88 / 0.57515.54 / 0.57815.97 / 0.591

4 sources. Left: MSE, right: MSE + SSIM.

SSIM goes up a little and PSNR goes down. The patch edges are gone, but the predictions are smoother. My hypothesis is that a blurry image with the right colors in the right places already gets a decent SSIM. A sharp edge in the wrong place would lower it, and so the model has no reason to predict sharp edges. The train PSNR is about as low as val, so the model is underfitting.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Ground truth

Prediction

3.2 Training and Evaluation

  "data_overrides": {
    "num_views": 8,
    "sampling_range": 32,
    "eval_num_frames": 32,
    "batch_size": 16,
    "num_workers": 8
  },
  "model": {
    "embed_dim": 384,
    "num_heads": 6,
    "num_layers": 8,
    "predict_points": false,
    "encoder_name": "dinov2_vitb14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 1000,
    "learning_rate": 0.0006,
    "weight_decay": 0.01,
    "min_lr_ratio": 0.05,
    "warmup_epochs": 10
  }

MetricValTest
Depth AbsRel0.1410.158
Depth δ10.8350.806
Rotation AUC @5 / @15 / @300.228 / 0.556 / 0.7160.199 / 0.526 / 0.734
Translation AUC @5 / @15 / @300.014 / 0.083 / 0.2010.012 / 0.074 / 0.192
Joint pose AUC @5 / @15 / @300.004 / 0.067 / 0.1760.002 / 0.050 / 0.174

09c1414f1b ground truth

09c1414f1b prediction

21d970d8de ground truth

21d970d8de prediction

280b83fcf3 ground truth

280b83fcf3 prediction

2b1dc6d6a5 ground truth

2b1dc6d6a5 prediction

The clouds appeared smeared along the viewing direction. The views that would have pinned those aspects down don't exist. The predicted cameras also seemed to be bunched closer than the ground truth cameras.

3.3 Bonus: Number of views

The model is trained with 8 views. How does it do with fewer or more views at test time?

ViewsAbsRelδ1Rotation AUC @5 / @15 / @30Translation AUC @5 / @15 / @30Pose AUC @5 / @15 / @30Pose pairsWall time (s)
20.1160.8610.264 / 0.590 / 0.7350.000 / 0.095 / 0.2130.000 / 0.074 / 0.185283.5
40.1270.8510.214 / 0.537 / 0.6990.023 / 0.068 / 0.1720.010 / 0.055 / 0.1481683.7
80.1410.8340.228 / 0.556 / 0.7160.014 / 0.083 / 0.2010.004 / 0.067 / 0.1767844.8
160.1740.7820.206 / 0.504 / 0.6750.011 / 0.066 / 0.1710.003 / 0.048 / 0.14533605.0
320.1920.7580.184 / 0.490 / 0.6760.013 / 0.070 / 0.1770.003 / 0.048 / 0.149138887.4

The depth gets worse with more views. Some of this is because more views cover more of the clip, but the jump above 8 views is larger, so the model also seems to do worse with more views than it was trained with. Presumably the global attention now sees a lot more tokens than in training. Pose is best around 8 views (the 2-view result is only 28 pairs, so it is noisy). The runtime grows with the number of views. Training with a random number of views should help.