Assignment 4: Multi-View Transformers

16-825: Learning for 3D Vision · Name: Zixin Guo · Andrew ID: zixinguo

Part 1 prepares canonicalized multi-view data from ScanNet++ iPhone frames (a4data/geometry.py). Part 2 trains Mini-LVSM, a decoder-only transformer that renders a new view from source images and camera rays (models/lvsm/). Part 3 trains Mini-VGGT, which predicts cameras and depth from RGB alone (models/vggt/).


1. Multi-View Data Preprocessing

1.1–1.5 What each function does

Visualizations

Points and cameras after unprojection (original world frame, meters). The red frustum is view 0, the black × is the valid-point centroid, and the dashed line connects camera 0 to the centroid.

unprojection scene 0

unprojection scene 2

RGB and depth of scene 0

Four of the eight sampled views of scene 036bce3393: resized RGB and the aligned z-depth (white = missing).

The same two scenes after canonicalization. Camera 0 sits at (0, 0, 0) and the centroid is exactly one unit away; the scene scale printed in the title is the centroid distance in meters before dividing.

canonicalization scene 0

canonicalization scene 2

check_data.py output

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
sequence views res valid depth max point reproj. (px) max ray reproj. (px) centroid distance
e050c15a8d_frame_005750 8 224 0.952 1.2e-4 1.3e-4 1.000
5f99900f09_frame_003000 8 224 0.992 9.9e-5 8.9e-5 1.000
47b37eb6f9_frame_007760 8 224 0.975 1.0e-4 8.2e-5 1.000
4c5c60fa76_frame_010770 8 224 0.989 1.1e-4 8.6e-5 1.000

Batch shapes (B = 2, V = 8): rgb (2,8,3,224,224), depth/depth_valid (2,8,1,224,224), K (2,8,3,3), T_world_from_camera (2,8,4,4), points_world/ray_origins/ray_directions (2,8,3,224,224), plucker (2,8,6,224,224), scene_scale (2,). All reprojection errors are around 1e-4 px, far under the 0.01 px limit. The full JSON is in data/check_data_train.json.

Short explanations

Canonicalization. Every training sample is rewritten in the coordinate frame of its first camera and rescaled so the valid-point centroid is one unit from that camera. Without this, the model would see the same room at arbitrary positions, orientations and scales, depending on where the original ScanNet++ world origin happened to be. With it, “camera 0 is at the origin looking down +Z, and the stuff it sees is about one unit away” holds for every sample, so the network learns one consistent convention. It also makes the depth and translation targets of Part 3 well scaled (around 1) rather than in meters that vary by room. The centroid is used for the scale only; it is not subtracted, since camera 0 must stay at the origin.

World-space ray directions. A pixel (u, v) sees along K⁻¹ (u, v, 1) in camera coordinates. Rotating by R (the camera-to-world rotation) gives the same direction in the canonical world frame; the translation does not apply to directions. After normalization, each pixel stores a unit vector that depends on both the intrinsics and the camera orientation, so the model can tell where a pixel looks in the shared frame without being handed R or K explicitly.

Plücker moment. m = o × d is the cross product of the ray origin and its direction. It is the same for every point on the line ((o + λd) × d = o × d), so [d, m] identifies the line itself, not just a point on it. Together with d it encodes the camera position: two rays with the same direction from different cameras have different moments. |m| is the distance from the world origin to the line, so for camera 0 (origin at 0) all moments are zero, which is a clean way to mark the reference view.


2. Novel-View Synthesis with Mini-LVSM

2.0 Implementation notes

2.1 Overfitting

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit (one fixed batch of 4 samples, S = 2, T = 6; 9.4 min on an A40).

Final overfit PSNR: 30.94 dB (MSE 8.6e-4, SSIM 0.871) on the memorized batch. The loss went from 0.063 to below 1e-3; the mean PSNR over all 2000 training steps was 25.4 dB.

LVSM overfit

val_epoch000.png after 2000 overfit steps. Columns: source view 1, source view 2, ground truth target, prediction; one row per target view. The prediction matches the target, including the two targets (rows 2–3) that look at a very different part of the room than the sources, which can only be memorized, not inferred.

2.2 Training and evaluation

Settings (configs/lvsm.json, unchanged): patch 8 (784 tokens per view), width 384, 8 blocks, 6 heads of 64, QK-norm, 14.6M parameters. S = 2 sources and T = 6 targets out of V = 8 views sampled from a 32-frame window, batch 4 (24 target images per step). AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps then cosine to 10 %, gradient clip 1.0, bf16 autocast. 100 epochs × 56 steps = 5600 steps; 2 h 16 min on one A40 (about 80 s per epoch, most of it data loading with 4 workers). Validation: 28 scenes, 8 views from the first 32 prepared frames, seed 825.

LVSM training curves

Training curves from log.jsonl. The validation PSNR was still going up at epoch 100 (16.13 dB, SSIM 0.560), so the model is limited by training time, not by overfitting. Train loss sits above val loss because the train numbers are averaged over an epoch while the weights are still moving, and because training windows come from anywhere in a scan while validation always uses the first 32 frames.

Validation metrics for every ablation (best.pt, 28 scenes; correct = true target rays, shuffled = rays of another target view, none = all-zero rays):

sources S conditioning PSNR (dB) SSIM MSE images
1 correct 15.19 0.558 0.0352 196
1 shuffled 11.80 0.487 0.0729 196
1 none 12.47 0.527 0.0589 196
2 correct 16.13 0.560 0.0294 168
2 shuffled 11.49 0.472 0.0778 168
2 none 12.45 0.527 0.0591 168
4 correct 16.87 0.575 0.0247 112
4 shuffled 11.59 0.473 0.0783 112
4 none 12.45 0.537 0.0588 112

For reference, three trivial predictors on the same validation targets (S = 2): copy source view 0 → 10.95 dB / 0.354; average of the two sources → 12.00 dB / 0.372; a constant gray image → 12.04 dB / 0.538.

LVSM ablation

epoch 0epoch 95

Training visualizations on the first validation batch (columns: 2 sources, GT, prediction). Left: epoch 0, the output is one flat color. Right: epoch 95, the layout (bright window, dark chair, table) is in the right place but everything is blurry at the 8 px patch level.

S=1 correct

S = 1, correct rays

S=4 correct

S = 4, correct rays

S=2 shuffled

S = 2, shuffled rays

S=2 none

S = 2, zero rays

Number of source views. More sources help every time: 15.2 → 16.1 → 16.9 dB for 1 → 2 → 4 sources, even though the model was trained with exactly 2. Each extra source covers more of the target’s field of view and gives the attention more patches to copy color from. The gain from 2 → 4 (0.7 dB) is smaller than from 1 → 2 (0.9 dB), while the sequence gets longer (3920 vs 2352 tokens), so the cost grows faster than the quality.

Ray conditioning. The target rays are the only thing that tells the model which view to render, and the ablation shows it uses them. With zero rays the output is the same for every target and PSNR drops to 12.45 dB, almost exactly the “mean of the sources” and “flat gray” baselines (12.0 dB): the model falls back to an average image. Shuffled rays are even worse (11.5 dB), below the gray baseline, because the model confidently renders the wrong view, and a sharp wrong image costs more than a blur. This is what a pose-conditioned renderer should do, and it shows that the Plücker tokens carry the camera pose and not just a “this is a target” flag (the target_proj bias alone would give the none result).

What limits quality. 16–17 dB is 4–5 dB above copying a source, but the renders are clearly blurry. The main reason is training time: 5600 steps with 24 target images each is very little for a transformer trained from scratch (the real LVSM trains far longer with much larger batches), and the validation PSNR was still rising. The second reason is the data: 8 views inside a 32-frame iPhone window often look at different parts of the room (copying a source only gives 11 dB), so many target pixels are not visible in any source, and MSE pushes those toward a blurry mean.

2.3 Bonus: patch size (model capacity vs. cost)

Question. Mini-LVSM spends most of its compute on attention over (S+1)·N tokens. Doubling the patch size from 8 to 16 cuts N from 784 to 196 (4× fewer tokens, ~16× cheaper attention) but each token must now encode a 16×16×9 patch and the head must decode a 16×16×3 patch. How much quality does this cost, and how much time and memory does it save, at a fixed training budget?

Setup. configs/lvsm_p16.json is identical to the baseline except patch_size: 16 (same width, depth, optimizer, schedule, 100 epochs, batch 4, S = 2, same seed). The patch-16 run had to be split into four sub-1-hour Slurm jobs with --resume (the queue only let me use the interactive partition), which does not change the schedule. Speed and memory come from tools/bench_lvsm.py, which times forward + backward on one synthetic training batch (B = 4, S = 2, T = 6, 224 px, bf16) on the same A40, so data loading is excluded.

Results (validation, best.pt of each run):

patch 8 (baseline) patch 16
parameters 14.6 M 16.0 M (bigger in/out projections)
tokens per sequence (S = 2) 2352 588
train step, model only 265 ms 63 ms (4.2× faster)
inference, one batch 84 ms 19 ms
peak training memory 8.9 GiB 2.5 GiB
wall-clock per epoch (data-bound) 80 s 104 s*
PSNR / SSIM, S = 1 15.19 / 0.558 15.01 / 0.548
PSNR / SSIM, S = 2 16.13 / 0.560 15.86 / 0.550
PSNR / SSIM, S = 4 16.87 / 0.575 16.46 / 0.562
PSNR, S = 2, shuffled / none rays 11.49 / 12.45 11.52 / 12.33

Both runs are limited by the 4-worker data loader (JPEG/PNG decoding plus undistortion for 32 frames per step), not the GPU, so the wall-clock does not reflect the model cost; the patch-16 run was even slower per epoch because it shared its nodes with other interactive jobs.

patch 8 vs 16 curves

Training curves of both runs. The two models track each other closely for the whole run; patch 8 pulls ahead by about 0.3 dB after epoch 40.

patch 8 epoch 95patch 16 epoch 95

Same validation batch at epoch 95: patch 8 (left) vs. patch 16 (right). The 16 px blocks are plainly visible on the right; the coarse layout is the same.

ablation for both patch sizes

Full ablation for both runs. The ray-conditioning pattern (correct ≫ none > shuffled) is identical for both patch sizes.

Conclusions.


3. Multi-View Geometry with Mini-VGGT

3.0 Implementation notes

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit: one training clip (036bce3393, 8 views), 40 epochs × 25 steps, lr 1e-3, 5 min on an A40.

VGGT overfit loss curve

Total, camera and depth loss while overfitting one clip. The total goes negative because the confidence term c·err − 0.2·log c is negative once the error is small and the confidence is large.

After 40 epochs on the fixed clip: camera loss 0.0026, depth loss −0.188, depth AbsRel 0.036, δ1 0.957, mean rotation error 0.17°, mean translation-direction error 0.35°, joint pose AUC@5° 0.986.

overfit GT vs prediction

GT (left) vs. cameras + depth predicted from RGB only (right) for the overfit clip, in canonical coordinates. The two clouds are visually identical; all eight frustums land on the GT ones.

overfit depth

RGB, GT depth and predicted depth for four of the eight views of the overfit clip (white = no GT depth, predictions are masked the same way).

3.2 Training and evaluation

Settings (configs/vggt.json defaults): frozen DINOv2 ViT-B/14, 8 frame/global pairs (16 blocks) of width 384 with 6 heads, camera head with 4 cross-view blocks, depth head with RGB refinement; 8 views from a 32-frame window, 224 px, batch 16, 16 data workers. AdamW lr 6e-4, weight decay 0.01, 10 warmup epochs then cosine to 5 %, gradient clip 1.0, bf16 autocast, loss = 5 × camera + 1 × depth. 1000 epochs × 14 steps = 14 000 steps; 8.7 h on one A40 (about 31 s per epoch). best.pt is epoch 874 (lowest validation loss).

VGGT training loss curve

Total, camera and depth loss (train and validation) over 1000 epochs. Both losses were still slowly decreasing at the end. The validation curve is below the training curve (train is averaged while the weights move, and random training windows are harder than the fixed first-32-frame validation windows), so there is no overfitting.

Metrics (best.pt, 8 views from the first 32 prepared frames, 224 px, seed 825; 28 scenes = 784 camera pairs per split, no degenerate pairs):

split Depth AbsRel ↓ Depth δ1 ↑ Rot. AUC @5/15/30° ↑ Trans. AUC @5/15/30° ↑ Joint AUC @5/15/30° ↑ mean rot. err mean trans. err
val 0.139 0.848 0.197 / 0.522 / 0.706 0.031 / 0.104 / 0.218 0.011 / 0.077 / 0.191 11.8° 39.3°
test 0.151 0.802 0.208 / 0.541 / 0.739 0.011 / 0.054 / 0.168 0.002 / 0.029 / 0.146 9.1° 40.8°

(Untrained model at epoch 0 on val: AbsRel 0.726, δ1 0.247, rotation AUC@30 0.286, translation AUC@30 0.0.)

GT vs. prediction on four validation scenes. Left: GT cameras and depth-unprojected points; right: the same views reconstructed from the predicted cameras and predicted depth. Below each: RGB, GT depth and predicted depth for four of the eight views.

val scene 0 3D

val scene 0 depth

09c1414f1b: living room. Layout, floor and far wall are right; the chairs become blocks and the predicted frustums are slightly rotated.

val scene 1 3D

val scene 1 depth

21d970d8de.

val scene 2 3D

val scene 2 depth

280b83fcf3: a narrow view with a lot of missing GT depth; the predicted cloud is wider and shallower than the GT.

val scene 3 3D

val scene 3 depth

2b1dc6d6a5: office. The chair and desk come out as coarse blobs, and the wall on the right is predicted closer than it is.

Errors on held-out scenes.

3.3 Bonus: number of input views at test time

Question. The model was trained with exactly 8 views. How do accuracy, runtime and memory change when the same weights are given 2, 4, 8, 12 or 16 views of a scene?

Setup. tools/vggt_view_sweep.py evaluates best.pt on the 28 validation scenes with V ∈ {2, 4, 8, 12, 16} views drawn from the first 32 prepared frames (frame 0 is always the reference; smaller sets are prefixes of the same random permutation, so the 2-view set is a subset of the 8-view set). Batch size 4 on one A40; time is wall-clock for the whole split, memory is peak allocated.

views Depth AbsRel ↓ Depth δ1 ↑ Rot. AUC @15/30° Trans. AUC @15/30° Joint AUC @15/30° pairs s / scene peak GPU mem
2 0.110 0.875 0.579 / 0.719 0.174 / 0.306 0.143 / 0.281 28 0.10 0.76 GiB
4 0.123 0.872 0.513 / 0.673 0.120 / 0.231 0.072 / 0.180 168 0.09 0.84 GiB
8 (trained) 0.139 0.848 0.522 / 0.706 0.104 / 0.218 0.077 / 0.191 784 0.15 1.00 GiB
12 0.147 0.815 0.501 / 0.691 0.088 / 0.197 0.061 / 0.170 1848 0.40 1.16 GiB
16 0.155 0.813 0.481 / 0.675 0.080 / 0.183 0.056 / 0.158 3360 0.79 1.32 GiB

Results and explanation.