Homework 4

Jacob Lessing

1. Multi-view Data Preprocessing

Unprojection

Canonicalization

Check Data Output

➜  assignment4_f26 git:(main) ✗ uv run python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.777836385183036e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011736395390471444,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanation

Canonicalization

The dataset picked an arbitrary origin for each of the samples. The neural network needs to choose an origin and a scale for its output, and that choice must be determined by the input, or otherwise the network output could be among infinitely many options and the loss won’t be well-defined. When we canonicalize the data, we transform all the camera frames (also depth data, and points) to be in the first camera’s frame, and rescale it so the distance from camera 0 to the valid-point centroid is 1. This makes it so the network can learn it’s always predicting a consistent thing: normalized scale in the first camera’s frame.

World-space Ray Directions

After canonicalization, we compute, for each pixel in each sample, a world-space ray direction: the unit vector from each camera view’s origin through that pixel (u, v), rotated to be in the canonical world frame.

Plucker Moment

The Plucker moment is a well-formed way of expressing these rays. The naive way of expressing a ray is by specifying the origin and its direction, d, but each ray has infinitely many of such descriptions (just pick any other point along the ray to be the origin). We instead use the moment m = o x d, because regardless of what point we choose as o the moment is the same. Then (d, m) uniquely describes the line.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

overfit PSNR: 30.92 dB columns: source views, ground truth, prediction

2.2 Training and Evaluation

Training Curves

Ablation Study

Source views (S) Conditioning PSNR (dB) SSIM MSE loss Target images
1 correct 15.19 0.5581 0.03516 196
1 shuffled 11.80 0.4872 0.07280 196
1 none 12.48 0.5281 0.05865 196
2 correct 16.13 0.5604 0.02942 168
2 shuffled 11.50 0.4724 0.07776 168
2 none 12.45 0.5266 0.05897 168
4 correct 16.88 0.5759 0.02460 112
4 shuffled 11.59 0.4733 0.07837 112
4 none 12.47 0.5364 0.05866 112

Prediction accuracy increases as the number of source views increases (PSNR goes up from 15.19 with S=1 to 16.88 with S=4). More sources better define the scene, meaning less must be infered by the network. Notably, the network was trained with S=2, so the fact that S=4 is better means the network not only generalizes to more views, but takes advantage of them.

Prediction accuracy is significantly higher when the network is given the correct target ray-conditioning rather than a shuffled one or none (with S=4, correct conditioning has PSNR 16.88, much greater than the shuffled 11.59 and none 12.47). Correct is better than no conditioning (none) which is better than confidently wrong (shuffled). This means the model is actually learning camera geometry rather than just giving some average scene.

Training, Evaluation Visualizations

Train Visualization (epoch 010)

columns: source views, ground truth, prediction

Train Visualization (epoch 060)

columns: source views, ground truth, prediction

Eval Visualization (S4, correct)

columns: source views, ground truth, prediction

Training Settings

Trained with S = 2 sources (6 targets), 8 views sampled from range of 32, image size 224 px, batch size 4, and 100 epochs.

{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "batch_size": 4,
    "eval_num_frames": 32
  },
  "model": {
    "patch_size": 8,
    "d_model": 384,
    "n_layers": 8,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": {},
  "training": {
    "epochs": 100,
    "learning_rate": 0.0002,
    "weight_decay": 0.05,
    "betas": [0.9, 0.95],
    "warmup_steps": 200,
    "grad_clip": 1.0,
    "amp": true,
    "vis_every": 5,
    "output_dir": "outputs/lvsm"
  },
  "num_source_views": 2
}

2.3 Bonus: Training View Count

Setup

Question: Does increasing the number of views in training to 4 increase novel-view prediction at test time?

Settings: Use identical settings to Q2.2 except change "num_source_views" in the config from 2 to 4.

Results

Evaluation

Test S PSNR (train S=2) PSNR (train S=4)
1 15.19 15.30
2 16.13 16.37
4 16.88 17.42

Training Curve

Eval Visualization

columns: source views, ground truth, prediction

Conclusion

Explanation

Training with 4 views rather than 2 consistently increases test time performance across the ablation over test samples. The improvement increases with larger Test S. Notably, the eval with Test S = 4 went from a PSNR of 16.88 to 17.42. This makes sense as training with more views teaches the network how to better take advantage of increased input view information. This increased performance was at the cost of longer training time (10s per epoch vs. previously only 8s)

Limitations

The S=4 run has fewer target views per step, confounding the effect on training of number of views and target views.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit Loss Curve

Comparing GT and Predicted Geometry

Ground truth Prediction

3.2 Training and Evaluation

Training Loss Curve

Validation/Test Metrics

Metric Validation Test
Depth AbsRel 0.146 0.154
Depth δ1 0.824 0.800
Rotation AUC@5 deg 0.223 0.216
Rotation AUC@15 deg 0.525 0.549
Rotation AUC@30 deg 0.690 0.743
Translation AUC@5 deg 0.016 0.020
Translation AUC@15 deg 0.080 0.086
Translation AUC@30 deg 0.200 0.196
Joint Pose AUC@5 deg 0.009 0.005
Joint Pose AUC@15 deg 0.066 0.057
Joint Pose AUC@30 deg 0.173 0.174

Comparing GT and Predicted Geometry

Scene Ground truth Prediction
1
2
3
4

Training Settings

This run used 8 views, batch size 16, 5.0 weight on camera loss, 1.0 weight on depth loss and 1000 training epochs.

{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "num_views": 8,
    "sampling_range": 32,
    "eval_num_frames": 32,
    "batch_size": 16,
    "num_workers": 8
  },
  "model": {
    "embed_dim": 384,
    "num_heads": 6,
    "num_layers": 8,
    "predict_points": false,
    "encoder_name": "dinov2_vitb14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 1000,
    "learning_rate": 0.0006,
    "weight_decay": 0.01,
    "output_dir": "outputs/vggt",
    "overfit_scenes": 1,
    "overfit_steps_per_epoch": 25,
    "min_lr_ratio": 0.05,
    "log_every": 10,
    "warmup_epochs": 10,
    "overfit_epochs": 40,
    "overfit_learning_rate": 0.001
  }
}

Discussion

Quantitatively, The model predicts depth and camera rotation moderately well, but seriously struggles predicting translation. On the held-out test set, the absolute relative depth metric of 15.4% means on average the depth estimate was off by about that much, which seems pretty good. Rotation AUC@30 deg on the test set is a relatively high 0.743 meaning the model predicts camera rotation pretty well too. Translation AUC@30 deg in the test set is a paltry 0.196, so the model really struggles here.

Qualitatively the scene visualization shows many key details are blurred which reflects the error in the depth measurement. In scene one and two the chairs are blended in to the background, and in scene 3 and 4 there’s an overall noisiness to the prediction. Further, many of the walls are doubled and offset, reflecting the error in the camera translation prediction.

3.3 Bonus: Evaluation View Count

Setup

Question: Does increasing the number of views at evaluation time improve depth and camera pose accuracy?

Settings: Use the same base model trained in 3.2 and identical evaluation settings to Q3.2 except vary --num-views to be 2, 4, 8, 16.

Results

Evaluation

Eval views Depth AbsRel Depth δ1 Rotation AUC@30 deg Translation AUC@30 deg Joint Pose AUC@30 deg Wall time (s)
2 0.108 0.891 0.725 0.273 0.249 3.01
4 0.127 0.858 0.656 0.172 0.141 3.37
8 0.146 0.824 0.690 0.200 0.173 4.26
16 0.167 0.786 0.681 0.182 0.161 6.13

Eval Visualization

Eval views Ground truth Prediction
2
4
8
16

Conclusion

Explanation

Suprisingly, more views don’t seem to help. With more eval views, the depth error is worse (increases from 0.108 @ V=2 to 0.167 @ V=16), the rotation AUC is worse (decreases from 0.725 @ V=2 to 0.681 @ V=16) and the translation AUC is worse (decreases from 0.273 @ V=2 to 0.182 @ V=16). Its possible V=2 did the best, because tith more cameras, the model has to learn their placement which is extra degrees of freedom making it a much harder problem.

Limitations

The 2-view metrics come from much fewer pairs than the 16-view metrics, so they are noisier.