16-825 Assignment 4: Multi-View Transformers

Data preprocessing · Mini-LVSM novel-view synthesis · Mini-VGGT cameras and depth. Everything here uses the ScanNet++ iPhone data at 224 px, and I ran all the experiments on RTX 3090s.

Part 1. Data Preprocessing

Unprojection

unprojection
All 8 depth maps unprojected using the resized intrinsics and the camera-to-world poses. The points from different views line up on top of each other in the same world frame, which is a good sign the geometry is right.

Canonicalization

canonicalization
After canonicalization the first camera (red frustum) sits right at the origin, center (0.0000, 0.0000, 0.0000). The scene scale for this sample is 1.867 m, and after dividing by it the valid-point centroid is exactly 1.000 away, as expected.

check_data.py output

split: train, dataset_samples: 224, checked_samples: 4
sample                      views  res  valid_depth  max_reproj_err(px)  max_ray_reproj_err(px)  centroid_dist
e050c15a8d_frame_007080       8    224    0.978         1.13e-4              1.09e-4               1.0000
5f99900f09_frame_002820       8    224    0.990         1.23e-4              1.11e-4               1.0000
47b37eb6f9_frame_002720       8    224    0.961         0.99e-4              0.89e-4               1.0000
4c5c60fa76_frame_006690       8    224    0.752         1.16e-4              1.07e-4               1.0000
tensor shapes: images [2,8,3,224,224], ray_directions [2,8,3,224,224], plucker [2,8,6,224,224]

The full JSON is in outputs/check_data.txt. Every reprojection error is around 1e-4 px, which is basically float32 round-off and far under the 0.01 px threshold.

Explanations

Part 2. Novel-View Synthesis with Mini-LVSM

This is the decoder-only version of LVSM. For each source view I stack the RGB (shifted to [-1, 1]) with its Plücker rays, so 9 channels, and patchify it. The target view only gets its Plücker rays (6 channels), since that's the image we're trying to predict. Both sides get a 2-D sin-cos position embedding. Every target view becomes its own sequence [all source tokens, target tokens] and goes through 8 pre-norm transformer blocks with QK-norm. At the end I take just the target tokens, run them through a linear head and a sigmoid, and unpatchify them back into an image. The loss is plain pixel MSE.

Q2.1 Overfitting

The overfit run ended at a validation PSNR of 30.92 dB (SSIM 0.870), so the model can clearly memorize a scene.

overfit
outputs/lvsm_overfit/val_epoch000.png. Columns: source views, ground truth, prediction.

Q2.2 Training and evaluation

Training settings. I kept the defaults from configs/lvsm.json:

I started with the default 100 epochs (0.7 h). When I looked at the curves they were still going up, and validation loss was below training loss, so the model was obviously underfitting. Because of that I trained the main model for 400 epochs instead (2.7 h on one RTX 3090), with everything else the same. I report both runs below. best.pt is the checkpoint with the lowest validation loss.

curves
Training curves (MSE, PSNR, val SSIM) for the 400-epoch run (blue) and the default 100-epoch run (orange). Validation always uses the same held-out scenes and frames, so it's smoother than the training curve. Those fixed frames also turn out to be a bit easier than random training samples, which is why val PSNR ends up above train PSNR.

Validation metrics for the 400-epoch model, PSNR (dB) / SSIM:

Source viewscorrect raysshuffled raysno rays (zeros)
115.89 / 0.58911.65 / 0.48212.41 / 0.531
217.17 / 0.59911.25 / 0.46012.37 / 0.528
418.35 / 0.62111.29 / 0.45612.38 / 0.539

And the default 100-epoch model, just for comparison:

Source viewscorrectshufflednone
115.19 / 0.55811.80 / 0.48712.47 / 0.529
216.13 / 0.56011.49 / 0.47212.43 / 0.528
416.87 / 0.57511.58 / 0.47312.44 / 0.538

Training visualizations (columns: source views, ground truth, prediction):

Epoch 0
Epoch 380

Evaluation visualizations (400-epoch model, columns: source views, ground truth, prediction):

S=1, correct rays
S=2, correct rays
S=4, correct rays
S=2, shuffled rays
S=2, no rays
S=1, shuffled rays
S=1, no rays
S=4, shuffled rays
S=4, no rays

Discussion.

Q2.3 Bonus: training view count

Question. If I change how many source views the model sees during training, how does that change its results when I evaluate with 1, 2 and 4 source views?

Setting. I trained three models that are identical except for num_source_views (1, 2 or 4): same default config, 100 epochs, same seed and schedule. Then I evaluated each one on the same validation set at S = 1, 2 and 4, using the correct rays.

Train S \ Eval S124train time
115.12 / 0.55914.90 / 0.55314.64 / 0.5610.52 h
215.19 / 0.55816.13 / 0.56016.87 / 0.5750.68 h
415.31 / 0.55616.41 / 0.56217.45 / 0.5800.87 h
Validation PSNR/SSIM vs. the number of source views at evaluation, one line per training setting (1, 2 or 4 views).
Training curves for the three models (1, 2 and 4 training views, 100 epochs each)
Model trained with S=1, evaluated at S=4
Model trained with S=4, evaluated at S=4

Findings.

Limitations. These are only 100-epoch runs (all of them still underfitting) and one seed each, so the small gaps at S=1 (≈0.2 dB) are within run-to-run noise. I didn't try sampling a random number of views per step, but I'd expect that to get the robustness of S=1 together with the fusion ability of S=4.

Part 3. Cameras and Depth with Mini-VGGT

Here's a quick overview of the model:

The losses are:

Q3.1 Overfitting a single scene

After 40 epochs on a single scene the camera loss is down to 0.0026. The total loss actually goes negative (−0.15), which is fine: it's just the −log(confidence) term once the model gets confident. The predicted geometry lines up with the GT, and the predicted centroid distance comes out at 0.999.

Overfit loss curves (total, camera, depth)
GT cameras + depth
Predicted cameras + depth

Q3.2 Training and evaluation

Training settings. Again I used the defaults, this time from configs/vggt.json:

For evaluation I use 8 views taken from the first 32 prepared frames, at 224 px with seed 825, over the full splits.

Loss curves for the normal training run (total, camera and depth; the components are shown unweighted)
SplitAbsRel ↓δ1 ↑Rotation AUC ↑Translation AUC ↑Joint pose AUC ↑
@5@15@30@5@15@30@5@15@30
val0.1540.8140.2450.5670.7350.0250.0850.1960.0110.0720.180
test0.1670.7700.2130.5360.7160.0140.0540.1570.0040.0340.137

Each split has 28 samples and 784 pose pairs, and none of the pairs are degenerate. The mean rotation error is 11.3° on val and 10.1° on test; the mean translation-direction error is 42.2° and 42.9°.

GT vs. prediction on four validation scenes (left is the GT cameras + depth, right is what the model predicted):

Scene 0, GT
Scene 0, prediction
Scene 1, GT
Scene 1, prediction
Scene 2, GT
Scene 2, prediction
Scene 3, GT
Scene 3, prediction

Discussion of held-out errors.

Q3.3 Bonus: number of input views

Question. How does the number of input views change accuracy, speed and memory?

Setting. I took the model trained with 8 views and evaluated it on the val split with 2, 4, 8 and 16 views, all drawn from the first 32 prepared frames (same seed 825, 224 px). For speed and memory, I timed a single forward pass (batch 1, bf16, RTX 3090, averaged after a warm-up) with anywhere from 2 to 32 views.

ViewsAbsRel ↓δ1 ↑Rot AUC@15Rot AUC@30Trans AUC@30Pose AUC@30pairs
20.1170.8570.6000.7460.2290.18728
40.1370.8510.5480.7170.1860.157168
8 (train)0.1540.8140.5670.7350.1960.180784
160.1850.7550.5160.7060.1670.1483360
Views248162432
ms / forward25.827.025.541.460.183.8
ms / view12.96.83.22.62.52.6
peak memory (GB)0.740.760.800.890.971.05
Accuracy, latency and memory vs. the number of input views

Findings.

Limitations. I only changed the number of views at evaluation time. Training with a random number of views (say 2–16) would probably fix the drop at 16 views, and that's the experiment I'd try next. Also, each view count is scored over a different set of pairs, so the numbers aren't strictly apples-to-apples.