16-825 Assignment 4

1. Multi-View Data Preprocessing

1.2 Depth Unproject

Scene 1: unproject

uv run python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000 --output outputs/unprojection

1.3 Camera and Depth Canonicalization

Scene 1: canonicalizaiton

uv run python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization

canonicalizaiton

1.7 Final Check

Canonicalization:

for V sampled views of the same scene, the first camera view, V[0] is used as the reference for the world, and all other camera frames in V are predicted relative to this frame.

World-space Ray Directions:

By default, the center of each camera serves as the origin for all its rays and each pixel has its corresponding direction. In World-space, the origin of the first camera view serves as the origin for all other camers, using the relative transforms calculated for canonicalization.

Plücker Moment

3D lines are usually represented by an origin o and direction d. But the same 3D line can be represented with a different origin o1 and the same direction d. This introduces a dependency on the choice of origin. To decouple our model training from this choice, we use Plücker moment, which is (o x d) perpendicular to both o and d. Any arbitrary choice of o will give the same Plücker moment.

check_data.py output:

uv run python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010746094631031156,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 9.281559323426336e-05,
      "max_ray_reprojection_error_px": 7.968755380716175e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 9.951626998372376e-05,
      "max_ray_reprojection_error_px": 8.800713840173557e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with mini-LVSM

2. 1 Overfitting

overfit

uv run python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit
Mini-LVSM: 14.6M params, device cuda, S=2, 2000 steps/epoch, 1 epochs
epoch   0 | train loss 0.00687 psnr 25.38 | val loss 0.00087 psnr 30.90 ssim 0.8698 | 1475s, targets

2.2 Training and Evaluation

Training Progress:

epoch 0 epoch 50 epoch 95
lvsm lvsm lvsm
uv run python -m models.lvsm.train --config configs/lvsm.json

Evaluation Progress:

1 2 4
Corrected lvsm lvsm lvsm
Shuffled lvsm lvsm lvsm
None lvsm lvsm lvsm

Validation metrics (epoch 100):

Source views (S) Conditioning PSNR (dB) SSIM MSE loss # images
1 Correct 15.20 0.5580 0.03515 196
1 Shuffled 11.81 0.4875 0.07277 196
1 None 12.48 0.5298 0.05873 196
2 Correct 16.14 0.5599 0.02936 168
2 Shuffled 11.49 0.4719 0.07785 168
2 None 12.45 0.5284 0.05903 168
4 Correct 16.86 0.5750 0.02478 112
4 Shuffled 11.61 0.4734 0.07809 112
4 None 12.47 0.5379 0.05863 112

In all 3 source view ablations, the correct source view gave the best performance, highlighting that the right ray bundle and Plücker moments are essential for learning good representation. Surprisingly None rays performed better than shuffled rays in PSNR, even though the images look more like a room in shuffled which is supported by SSIM. I believe the reason for this is that in None, all the rays now just attend to each other equally because every one of them is the same and just even out the final output through out the image.

The other takeaway is the improved performance from increased number of views. The model is able create a more accurate 3D representation in its latent and informing the desired view synthesis.

2.3 Smaller Patch Size

Because of Time and GPU constraints, I am evaulating the smaller patch size model primarily for the 25th epoch. From the regular patch size of 8, we get:

epoch  25 | train loss 0.03866 psnr 14.77 | val loss 0.03552 psnr 14.95 ssim 0.5457 | 33s

each epoch takes about 31 seconds GPU memory: 9904 mb PSNR (4 Views) : 15.71 SSIM (4 Views) : 0.5558 Loss (4 Views) : 0.030

Running a smaller patch size model was not possible on my GPU with 16GB VRAM, so I had to use grad check and recompute all the forward values during my backward pass.

each epoch takes about 221 seconds (with grad checkpointing) GPU memory: 9896 mb (with grad checkpointing) PSNR (4 Views) : 15.07 SSIM (4 Views) : 0.5796 Loss (4 Views) : 0.034

p = 4 p = 8
p4 p8

While p = 4 is not as blocky as p = 8, it does seem like it takes a lot longer for it to train, as the loss for patch size 8 reduced more than patch size 4.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

overfit

Ground Truth Prediction
p4 p8
uv run python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

3.2 Training and Evaluation

This model was trained for 850 epochs (didn't have more time to train it). The rest of the seeting were default values with 8 views and sampling_range = 32

train

Ground Truth Prediction
p4 p8
p4 p8
p4 p8
p4 p8
uv run python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit --850

uv run python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/metrics.json --visualization outputs/vggt_train/visualization/index.html

uv run python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt --split test --eval-root data/scannetpp/prepared_eval --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 --report outputs/vggt_train/test_metrics.json

Validation/Test Metric

Metric 3 deg 5 deg 15 deg 30 deg
Rotation AUC 0.0689 0.1497 0.4052 0.5812
Translation AUC 0.0072 0.0148 0.0510 0.1351
Joint Pose AUC 0.0017 0.0074 0.0355 0.1086
Metric Value
Depth AbsRel 0.1749
Depth δ1 0.7611
Total loss 1.0304
Camera loss 0.1398
Depth loss 0.3317

The model is doing good in terms of rotation, that's why the points are probably in the right space to begin with/ But the translation and joint pose AUCs are not looking great. More than 82% of them have an error greater than 30 deg.

The depth prediction has done pretty well.