Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision · Yongyi Xiong (andrewid: yongyix)

Compute. Part 1, Q2.1 and Q3.1 ran locally on an Apple M4 Max (Q2.1 on MPS in fp32, Q3.1 on CPU because the provided VGGT metrics use float64). Full training for Q2.2, Q2.3 and Q3.2 and the timing/memory runs of Q3.3 ran on a single NVIDIA A100 40 GB in Google Colab with bf16 autocast. The Q3.3 window experiment ran locally on CPU in fp32. Apart from the TODOs, the only code changes are a per-step sampled source count in the LVSM training loop for Q2.3 and a separate evaluation script for Q3.3; the provided training, evaluation and metric code is otherwise unchanged.

1. Multi-View Data Preprocessing

1.1–1.3 Unprojection and Canonicalization

The same 8 frames of each scene are shown in the original world frame (meters, after Q1.1–1.2) and after canonicalization (Q1.3). In the canonicalized view the first camera (red frustum) sits at the origin with identity pose, and the side panel reports a valid-point centroid distance of 1.000 normalized units.

unprojection 079a326597
Scene 079a326597, unprojection, original world frame (m). First camera center (3.756, 2.082, 1.533) m.
canonicalization 079a326597
Scene 079a326597, canonicalization. First camera center (0, 0, 0); scene scale 2.080 m; valid-point centroid distance 1.000.
unprojection 07ff1c45bb
Scene 07ff1c45bb, unprojection, original world frame (m).
canonicalization 07ff1c45bb
Scene 07ff1c45bb, canonicalization. First camera at the origin; centroid distance 1.000.

In both stages the point clouds from different views overlap without visible double surfaces, which indicates that intrinsics, z-depth unprojection and camera-to-world poses are consistent.

1.7 Final Check

Output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 (full text: check_data_output.txt):

SampleViewsResolutionValid depth Max point reproj. error (px)Max ray reproj. error (px)Centroid distance
e050c15a8d_frame_007080822497.8%1.11e-49.65e-51.0000
5f99900f09_frame_002820822499.0%1.11e-41.08e-41.0000
47b37eb6f9_frame_002720822496.1%1.07e-48.45e-51.0000
4c5c60fa76_frame_006690822475.2%1.14e-49.86e-51.0000
batch shapes (batch size 2):
rgb, ray_origins, ray_directions, points_world   (2, 8, 3, 224, 224)
depth, depth_valid                               (2, 8, 1, 224, 224)
plucker                                          (2, 8, 6, 224, 224)
K (2, 8, 3, 3)   T_world_from_camera (2, 8, 4, 4)   original_world_from_reference (2, 4, 4)
frame_numbers (2, 8)   scene_scale (2,)

All reprojection errors are about 1e-4 px, far below the 0.01 px threshold, and every centroid distance is 1 up to float32 rounding.

Explanations

Canonicalization

Every sample is expressed in the coordinate frame of its first sampled camera. With camera-to-world poses Ti, the new poses are T0−1 Ti, so camera 0 becomes the identity (computed in float64 and cast back to float32). Depth is unprojected in this frame, and the scene scale s is the distance from camera 0 to the centroid of all valid points of the selected views. Translations, depth and points are divided by s; rotations are unchanged and the centroid is not subtracted, so camera 0 stays at the origin.

T̂i = T0−1 Ti,   s = ‖mean(pvalid)‖,   t̂i = ti / s,   ẑ = z / s,   p̂ = p / s

The original world frame of a ScanNet++ scan is arbitrary and its metric size varies from room to room, while the images alone determine geometry only up to a rigid transform and a global scale. Canonicalization removes exactly these unobservable degrees of freedom, so the regression targets are consistent across samples and the model never has to guess where the dataset's world origin was.

World-space ray directions

For pixel center (u, v), the camera-space direction is K−1[u, v, 1]T. It is rotated into the canonical world frame with the camera-to-world rotation and normalized to unit length; the ray origin is the camera center t, shared by every pixel of a view:

d = normalize(R K−1 [u, v, 1]T),   o = t

The intrinsics come from Q1.1. Because the provided resize uses align_corners=False, the mapping is xout + 0.5 = (sizeout/sizein)(xin + 0.5), so the focal lengths are scaled by the size ratio and the principal point becomes c' = r (c + 0.5) − 0.5. Since t is in canonical units, the rays live in the same frame and scale as the depth targets.

Plücker moment

Each ray is stored as six channels [d, m] with moment m = o × d. The moment does not depend on which point of the line is used as the origin, because (o + λd) × d = o × d, so (d, m) describes the line itself. m is perpendicular to the plane that contains the line and the world origin, and since d is a unit vector, ‖m‖ equals the distance from the origin to the line. The direction alone tells the model where a pixel looks; the moment adds where the ray is located, which distinguishes parallel rays from different camera centers. Every ray of camera 0 passes through the origin, so its moment is zero.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

One fixed batch (batch size 2, 2 source views, 6 targets per sample) was fitted for 2000 steps (about 34 minutes on MPS). The final overfit validation PSNR is 34.04 dB (SSIM 0.915), above the 30 dB target.

overfit val_epoch000
val_epoch000.png after overfitting. Columns: source views, ground truth, prediction.

2.2 Training and Evaluation

Training settings

LVSM training curves
Training curves from log.jsonl.

Train and validation PSNR stay close throughout (train 15.46 vs val 16.07 at epoch 90), so the model is not overfitting; the validation curve flattens after roughly epoch 75.

Ablation results (validation)

Source views SConditioningPSNR (dB)SSIMTarget images
1correct15.190.556196
1shuffled11.750.485196
1none12.510.529196
2correct16.070.558168
2shuffled11.400.470168
2none12.510.526168
4correct16.780.574112
4shuffled11.550.475112
4none12.520.535112

Training visualizations

epoch 0
Epoch 0
epoch 50
Epoch 50
epoch 90
Epoch 90 (best)

Columns in every LVSM visualization: source views, ground truth, prediction.

Evaluation visualizations

S1 correct
S = 1, correct
S2 correct
S = 2, correct
S4 correct
S = 4, correct
S2 shuffled
S = 2, shuffled
S2 none
S = 2, none

All nine settings: S1 correct / shuffled / none; S2 correct / shuffled / none; S4 correct / shuffled / none.

Discussion

Number of source views. With correct rays, PSNR rises from 15.19 (S = 1) to 16.07 (S = 2) and 16.78 (S = 4), even though the model was trained only with S = 2. More sources cover more of the room, so a larger fraction of each target can be copied from a nearby view instead of being hallucinated. The gain from 2 to 4 is smaller than from 1 to 2, and SSIM changes much less than PSNR (0.556 to 0.574): the extra views mostly fix colors and coarse layout, while fine structure stays blurry.

Ray conditioning. Removing the target rays (none) drops PSNR to about 12.5 dB for every S. Without target-view conditioning, the predictions become a much blurrier, nearly constant image close to the scene's average colors. They still depend on the source RGB and source rays, so changing S can still affect the output, even though the PSNR here barely moves (12.51 to 12.52 dB). Shuffled rays are worse than no rays (11.4 to 11.8 dB) because the model confidently renders a plausible but wrong viewpoint, and a sharp wrong image is penalized more by MSE than a blurry average. The large gap between correct and shuffled (3.4 to 5.2 dB) shows that the model really uses the target Plücker rays.

Failure modes. Even with correct rays the predictions are blurry and show visible 8×8 patch blocks, because each patch is decoded independently by a linear head and the MSE loss favors averaging when the target content is uncertain. Regions that no source view observes are filled with smooth color.

2.3 Bonus: Training with a Random Number of Source Views

Question

The baseline is always trained with S = 2 but is evaluated with S = 1, 2 and 4. Does training with a random number of source views per step, S ∈ {1, 2, 3, 4}, make the model better at the S values it was not trained on, and does it cost anything at S = 2?

Settings

Results

SConditioningBaseline PSNRRandom-S PSNR ΔPSNR (paired)95% CIBaseline SSIMRandom-S SSIM
1correct15.1915.16−0.03[−0.08, +0.03]0.5560.559
1shuffled11.7511.81+0.05[+0.02, +0.09]0.4850.490
1none12.5112.42−0.09[−0.16, −0.02]0.5290.534
2correct16.0716.07+0.00[−0.05, +0.06]0.5580.559
2shuffled11.4011.47+0.07[+0.04, +0.10]0.4700.473
2none12.5112.39−0.12[−0.20, −0.05]0.5260.531
4correct16.7816.87+0.09[+0.02, +0.19]0.5740.572
4shuffled11.5511.53−0.02[−0.07, +0.03]0.4750.470
4none12.5212.42−0.11[−0.20, −0.01]0.5350.541
validation curves comparison
Validation PSNR/SSIM (S = 2) during training, baseline vs random S.
paired delta psnr
Per-image PSNR difference (random S minus baseline) on identical validation targets, with the mean and 95% scene-bootstrap CI.
random S training curves
Random-S training curves (train PSNR mixes S = 1 to 4).
random S epoch 95
Random-S validation visualization, epoch 95 (S = 2).

Evaluation images for the random-S model: S1 correct · S2 correct · S4 correct · S2 shuffled · S2 none.

Conclusions

Random-S training gives small, mixed changes and no clear benefit. All nine settings differ from the baseline by at most 0.12 dB PSNR and 0.005 SSIM, the S = 2 validation curves overlap for the whole run, and the predictions look the same. Several paired CIs exclude 0, but in both directions: S = 4 with correct rays (+0.09 dB, CI [+0.02, +0.19]) and shuffled rays at S = 1 and 2 (+0.05 and +0.07 dB) are slightly better, while none is about 0.1 dB worse at every S. At S = 1 with correct rays, the random-S model, which saw S = 1 in a quarter of its steps, is not better (−0.03 dB, CI includes 0). The none rows cannot serve as a pure noise reference: without target rays the source RGB, the source rays and the number of tokens still change with S, and the two models were trained differently. Each configuration was trained once with one seed, so the CIs only cover variation over scenes and images, not over retraining, and I cannot tell whether these small differences would survive another training run.

A likely explanation is that the baseline already generalizes across S. Each target token attends to all source tokens with full attention, so adding or removing a source view only changes the length of the attention context, and the baseline's PSNR at S = 1 and 4 already follows the expected trend. With this small model and a 100-epoch budget, capacity and blur seem to limit quality more than the training view count does.

Limitations. One seed per configuration; best.pt is selected at S = 2 only; the sampled S range stops at 4, so S > 4 was not tested; and a longer schedule might separate the runs.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

The default model (frozen DINOv2 ViT-B/14, 8 frame/global attention pairs, width 384, 6 heads) was fitted to one fixed training clip of 8 views for 40 epochs of 25 steps (1000 steps, lr 1e-3, about 19 minutes on CPU).

Total lossRotation err. (°)Translation err. (°) AbsRelδ1Pose AUC@5
Before training5.4949.790.00.6190.2080.00
After 1000 steps−0.190.120.170.0360.9551.00

The loss becomes negative because the confidence term contains −0.2·log(confidence), which keeps decreasing as the model grows more confident on a clip it has memorized.

overfit loss curve
Overfit loss curve (total, camera and depth loss).
overfit GT vs prediction
Ground-truth geometry (left) vs geometry reconstructed from the predicted cameras and depth (right), with the view-0 RGB, GT depth and predicted depth.

The predicted cameras land on the GT cameras and the reconstructed room matches the GT point cloud. The predicted depth is slightly blocky at the 14×14 patch scale, since the coarse depth is decoded per patch before RGB refinement.

3.2 Training and Evaluation

Training settings

VGGT loss curve
Normal-training loss curve (total, camera and depth loss).

The validation loss is below the training loss for the whole run (0.82 vs 0.95 at epoch 985). Training batches use a random 32-frame window and random view order, while validation uses fixed views that always start at frame 0, so the two losses are not measured on the same distribution; I did not run an experiment to confirm which difference causes the gap.

Metrics

SplitAbsRel ↓δ1 ↑ Rot AUC @5/15/30Trans AUC @5/15/30Pose AUC @5/15/30 Mean rot err.Mean trans err.
Val0.1340.8400.178 / 0.499 / 0.696 0.019 / 0.083 / 0.1960.005 / 0.064 / 0.17512.3°41.1°
Test0.1520.8110.198 / 0.529 / 0.723 0.013 / 0.073 / 0.1800.003 / 0.055 / 0.1639.6°39.3°

28 scenes and 784 view pairs per split, no degenerate pairs. Raw reports: val metrics.json · test_metrics.json.

GT vs predicted geometry on four validation scenes

In each panel: GT cameras and points (top left), points reconstructed from predicted cameras and depth (top right), and the view-0 RGB, GT depth and predicted depth (bottom). The first camera is red.

val scene 09c1414f1b
Scene 09c1414f1b
val scene 21d970d8de
Scene 21d970d8de
val scene 280b83fcf3
Scene 280b83fcf3
val scene 2b1dc6d6a5
Scene 2b1dc6d6a5

Errors on held-out scenes

3.3 Bonus: Number and Spacing of Input Views

Question

How does the number of input views V affect accuracy, runtime and memory at evaluation time? In particular, do more views help the weak translation estimates from Q3.2? A follow-up experiment asks whether the result is driven by the number of views or by how far apart the views are.

Settings

Experiment A: number of views

view count val
Validation. Solid: all views; dashed: first 4 views. Time and memory axes start at 0.
view count test
Test, same layout.
SplitVGroupPairsAbsRelAbsRel aligned δ1Rot AUC@30Trans AUC@30Pose AUC@30 Forward (ms)Peak (MiB)Extra (MiB)
val2all280.1220.0850.8530.7100.1850.17642.358819
val4all = first 41680.1270.0890.8460.6630.1580.14442.860535
val8all7840.1340.0930.8400.6970.1950.17542.564877
first 41680.1360.0880.8320.6810.1780.152
val16all33600.1660.0960.7930.6720.1820.15642.8729154
first 41680.1700.0890.7980.7010.1670.149
val32all138880.1740.0970.7850.6640.1750.14944.7888303
first 41680.1780.0900.7920.6820.1640.141
test2all280.1220.0930.8660.7350.1850.11842.758819
test4all = first 41680.1400.1000.8240.7460.1900.15842.860535
test8all7840.1520.0970.8110.7230.1810.16343.164877
first 41680.1540.0990.8060.7380.2100.182
test16all33600.1620.0950.7920.7260.1830.16743.5729154
first 41680.1640.0980.7870.7180.1960.167
test32all138880.1680.0950.7900.7440.1790.16543.8888303
first 41680.1700.0990.7900.7190.2010.172

The V = 8 all-views row matches the official Q3.2 evaluation (AbsRel and δ1 exactly, AUC within 0.001), which checks the script. Full tables: val csv · test csv.

  1. More views make raw depth worse, but almost only through scale. Raw AbsRel rises from 0.122 (V = 2) to 0.174 / 0.168 (V = 32) on val / test, and the same happens on the fixed first 4 views (0.127 to 0.178 on val). After per-scene scale alignment, the first-4-view AbsRel stays flat (0.088 to 0.090 on val, 0.098 to 0.100 on test). Scale-aligned depth error remains nearly constant on the fixed views; what changes most is the absolute scale of the prediction relative to the GT scale. The mean |log a| of the alignment factor on the first 4 views grows from 0.082 at V = 8 to 0.113 at V = 32 on val, driven by a few scenes whose factor approaches 0.5.
  2. More views do not fix translation. Translation AUC@30 stays between 0.16 and 0.21 for every V and both groups, and rotation AUC@30 between 0.66 and 0.75, with no monotonic trend. I did not compute CIs for Experiment A, and the first-4-view group has only 168 pairs, so small differences between V should be read with caution; what matters here is that no improvement is consistent across V, groups and splits. The hypothesis that extra context views would improve camera centers is not supported.
  3. Runtime is flat and memory grows linearly. At batch size 1 the forward pass takes 42 to 45 ms for every V. I did not profile the forward pass; one possible explanation is that with at most 32 × 256 tokens the A100 is underused and fixed per-kernel costs dominate, which would hide the growth in compute. The extra peak memory roughly doubles with V (19, 35, 77, 154, 303 MiB), so it is linear in V in this range. This is consistent with an attention kernel that does not store the full attention matrix (the model calls F.scaled_dot_product_attention), but I did not check which kernel was used. Linear memory does not mean linear compute: the FLOPs of global attention still grow quadratically with the total token count. Total peak memory goes from 0.57 to 0.87 GiB, well within the GPU.

Experiment B: view spacing (V = 8)

In Experiment A the window is fixed at 32 frames, so a larger V also means views that are closer together. To test whether translation fails because neighboring views have too little parallax, Experiment B keeps V = 8 and changes how far apart the views are. Changing W also changes which frames are sampled, so the images, the scene content they cover and the camera pairs being scored all change together; parallax is not the only variable. The table also lists the median GT distance between camera centers over all pairs and the median relative rotation.

window ablation
V = 8, eval window 16 to 128 prepared frames (val solid, test dashed). The dotted line marks the training window.
SplitWMedian baseline (m)Median rel. rotation AbsRelAbsRel alignedRot AUC@30Trans AUC@30 Δ Trans AUC@30 vs W = 32 [95% CI]Pose AUC@30
val160.1416.8°0.1110.0880.7820.123−0.072 [−0.122, −0.025]0.114
val320.2926.7°0.1340.0930.6970.194reference0.175
val640.4441.2°0.2220.1050.5080.188−0.006 [−0.048, +0.033]0.129
val1280.8157.1°0.3770.1210.3520.121−0.073 [−0.119, −0.028]0.061
test160.1011.3°0.1130.0890.8400.112−0.070 [−0.118, −0.023]0.102
test320.2223.3°0.1520.0970.7230.181reference0.163
test640.3932.7°0.2280.1040.6150.155−0.027 [−0.070, +0.015]0.127
test1280.6354.8°0.3250.1160.3840.160−0.021 [−0.065, +0.021]0.100

The CI is a scene-level bootstrap of the paired per-scene difference (every scene has the same 28 pairs). Full table: window_summary.csv.

  1. A narrower window hurts translation. Halving the window to W = 16 (median baseline 0.10 to 0.14 m) lowers translation AUC@30 by about 0.07 on both splits, and the CIs exclude 0. This is consistent with too little parallax: when the cameras barely move, the translation direction is hard to infer from the images. Since the sampled images and pairs also change with W, this experiment does not isolate parallax as the cause.
  2. A wider window does not help. At W = 64 and 128 the baselines are 2 to 3 times larger, yet translation AUC does not rise above the W = 32 value (the paired CIs include 0 at W = 64 on both splits and at W = 128 on test), and rotation AUC falls steadily (0.70 to 0.35 on val). Raw AbsRel triples while aligned AbsRel grows much less (0.093 to 0.121 on val), so most of the depth loss is again scale. Possible reasons are that the model never saw views this far apart during training and that wide windows have less overlap between views; I did not test either one separately.

Conclusions

For this model, the input that works best is the one it was trained on: 8 views spread over a 32-frame window. Adding views at test time keeps the scale-aligned depth error and the camera accuracy roughly unchanged, but makes the absolute scale less reliable, so raw AbsRel gets worse. It costs very little: runtime is flat and memory grows linearly up to 32 views. Translation, the main weakness from Q3.2, was not improved by either test-time change I tried. More views do not help, a narrower window makes it worse, and a wider window does not help and moves the input out of the training distribution. Improving it likely requires changes during training, for example training with more and wider views or a stronger camera head.

Limitations