16-825 Assignment 4: Multi-View Transformers

Yi Zhao · Andrew ID yizhao5

Everything below was produced by the code in the submitted zip, run on single GPUs of the lab cluster (RTX PRO 5000 for Mini-LVSM, A100 for Mini-VGGT) on the full course data (224 training scenes; the prepared validation and test splits with 28 scenes each). Numbers in the text are read from the run outputs. The cluster's containers have a 64 MB /dev/shm, too small for PyTorch DataLoader worker processes, so batches are loaded by threads instead (a4data/threaded_loader.py, opt-in); this changes only how fast samples are produced, not which samples or how they are computed.

1. Multi-View Data Preprocessing

All five Q1 functions are implemented in a4data/geometry.py (vectorised, no loops). resize_intrinsics applies xout = s(xin + 0.5) − 0.5 per axis as a matrix A K; unproject_depth scales K-1[u, v, 1] by z-depth and maps it with the camera-to-world pose; canonicalize_cameras uses T0-1 Ti in float64 and the provided scene scale; camera_rays and plucker_coordinates follow the data contract. As an independent check, recomputing rays, Plücker coordinates and unprojected points from the TA-prepared validation cameras reproduces the prepared fields to 2.4e-7.

Visualizations after unprojection (original world frame, meters)

scene 036bce3393: cameras (red = first) and unprojected points in ScanNet++'s world frame
scene 079a326597: the first camera sits at (3.76, 2.08, 1.53) m

Visualizations after canonicalization

scene 036bce3393 in camera 0's frame
scene 079a326597: first camera center (0, 0, 0), scene scale 2.080 m, valid-point centroid distance 1.000 (text panel, lower right)

Same frames in both stages (fixed sampling, seed 825). The right-hand panel of each viewer prints the first camera center and the valid-point centroid distance.

check_data.py on four training samples

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4

sampleviewsres.valid depthmax point reproj. [px]max ray reproj. [px]centroid dist.
260fa55d50_frame_00106082240.8791.15e-048.22e-051.0000
f6659a3107_frame_00501082240.9661.11e-041.03e-041.0000
8be0cd3817_frame_00513082240.9871.10e-041.02e-041.0000
a980334473_frame_00308082240.9128.63e-057.63e-051.0000

Full output of the command (JSON printed by check_data.py):

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "260fa55d50_frame_001060",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.8790407776832581,
      "max_reprojection_error_px": 0.00011475836799945682,
      "max_ray_reprojection_error_px": 8.217109279939905e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "f6659a3107_frame_005010",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9656234979629517,
      "max_reprojection_error_px": 0.00011108566104667261,
      "max_ray_reprojection_error_px": 0.00010327453492209315,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "8be0cd3817_frame_005130",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9871004223823547,
      "max_reprojection_error_px": 0.0001100326917367056,
      "max_ray_reprojection_error_px": 0.00010235906665911898,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "a980334473_frame_003080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9118602275848389,
      "max_reprojection_error_px": 8.63167442730628e-05,
      "max_ray_reprojection_error_px": 7.62939453125e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

All reprojection errors are about 1e-4 px (limit 0.01 px), every centroid distance is 1.0 and all shapes match the data contract.

Canonicalization

Each training sample is a set of V views of one scene, with camera-to-world poses in ScanNet++'s world frame (meters). That frame is arbitrary from the network's point of view: the same room can sit anywhere and at any orientation, and rooms differ in size. Canonicalization removes both nuisances. All poses are re-expressed relative to the first sampled camera, T'i = T0-1 Ti (computed in float64), so camera 0 becomes the identity and every other camera is described relative to it. Depth is then unprojected into this frame, and the distance from camera 0 to the centroid of all valid points across all views, s = ||mean(pvalid)||, is divided out of every translation, depth and point. After this step camera 0 is at the origin looking down +Z and the scene's centroid lies at distance 1. The scale is only a division, without subtracting the centroid, so camera 0 stays at the origin; rotations are unaffected and z-depth scales by the same factor as the points, so the depth/point/camera relations stay exact.

World-space ray directions

The ray through pixel (u, v) starts at the camera center and has camera-frame direction K-1[u, v, 1]T. Its z component is 1, so multiplying it by z-depth gives the 3D point, which is how unproject_depth uses it. Rotating by the camera-to-world rotation, d = R K-1[u, v, 1]T / ||·||, expresses the ray in the shared canonical frame, and the origin is the camera center o = t. Expressing every view's rays in one common frame tells the network how the views relate geometrically; camera-frame rays would be identical for every camera with the same intrinsics and would carry no pose information.

Plücker moment

A ray is a line, and a point plus a direction over-parameterizes it, since any point on the line gives the same ray. Plücker coordinates (d, m) with the moment m = o × d fix that. Moving the origin along the ray, o + λd, leaves the moment unchanged because d × d = 0, so the 6-vector depends only on the line itself. The moment is perpendicular to both d and the line's position vector, so m · d = 0 (the check in validate_sample). Its norm equals the distance from the world origin (camera 0) to the line, because d has unit length. For the transformer, this gives a smooth, translation-aware, per-pixel encoding of where each pixel's ray lies in the canonical frame: all six channels change smoothly across the image and between cameras, unlike raw 4×4 extrinsics given once per image.

2. Novel-View Synthesis with Mini-LVSM

Implementation (Q2.0). patchify reshapes (L, C, gh, p, gw, p) and permutes to (L, gh, gw, p, p, C), so patches are row-major and each is flattened in (row, column, channel) order; unpatchify applies the inverse permutation (a unit test checks exact round-trips and the position of individual pixels). build_tokens adds the shared 2D sine-cosine embedding to source and target tokens, repeats the S·N source tokens for every target and appends that target's N query tokens, giving (B·T, (S+1)·N, D). forward projects [2·RGB − 1, Plücker] source patches with source_proj and target Plücker patches with target_proj, runs input_norm and the blocks, keeps the last N tokens of each sequence and decodes them with out_norm, rgb_head, a sigmoid and unpatchify. A CPU test confirms that each target's output depends only on its own rays and on the sources of its own sample. The loss is the mean squared error over every pixel and view; the training step runs the model under bfloat16 autocast and casts the predictions to float32 before the loss.

2.1 Overfitting

Final overfit PSNR = 32.14 dB (SSIM 0.884)

2000 steps on one fixed training batch (4 samples × 8 views, S = 2, T = 6), configs/lvsm.json otherwise; the PSNR averaged over all 2000 steps is 26.82 dB, the final evaluation of the same batch 32.14 dB, so the loss decreased throughout.

val_epoch000.png of the overfit run: columns = source views, ground truth, prediction

2.2 Training and evaluation

Settings (configs/lvsm.json unchanged): 14.6 M parameters (patch 8, width 384, 8 blocks, head width 64, QK-norm); 8 views per sample, S = 2 sources (evenly spaced in the random view order) and T = 6 targets; batch 4; 100 epochs over the 224 training scenes (one random 8-view window per scene and epoch, 56 steps per epoch, 5600 steps); AdamW, peak learning rate 2e-4 after 200 warm-up steps, cosine decay to 10%, weight decay 0.05, betas (0.9, 0.95), gradient clipping 1.0, bfloat16 autocast. One RTX PRO 5000, 31 minutes. Validation after every epoch uses the prepared validation split; best.pt is the epoch with the highest validation PSNR.

training curves from log.jsonl: MSE (log scale), PSNR and validation SSIM per epoch

Best checkpoint: epoch 98 of 100 (validation PSNR 16.11 dB, SSIM 0.5574).

Validation (28 scenes, 8 views from the first 32 prepared frames, seed 825)

Starget raysPSNR [dB]SSIMtarget images
1correct15.170.5562196
1shuffled11.790.4852196
1none12.410.5275196
2correct16.110.5574168
2shuffled11.450.4692168
2none12.390.5224168
4correct16.830.5725112
4shuffled11.550.4717112
4none12.430.5327112

Test (28 scenes, same protocol)

Starget raysPSNR [dB]SSIMtarget images
1correct15.500.5339196
1shuffled12.350.4684196
1none12.630.5065196
2correct16.160.5388168
2shuffled12.160.4600168
2none12.640.5077168
4correct16.980.5392112
4shuffled11.790.4441112
4none12.640.5041112

Additional validation points, correct rays: S = 3: 16.65 dB / 0.5608; S = 6: 16.65 dB / 0.5400 (only 2 targets per scene).

Training visualizations (validation batch; columns: source views, ground truth, prediction)

epoch 1
epoch 51
epoch 96

Evaluation visualizations (best checkpoint; columns: source views, ground truth, prediction)

All nine evaluation settings (first validation batch, up to four target views per picture; rows of each picture are target views). Rows below: S = 1, 2, 4; columns: correct, shuffled and zero target rays.

S = 1, correct target rays
S = 1, shuffled target rays
S = 1, zero target rays
S = 2, correct target rays
S = 2, shuffled target rays
S = 2, zero target rays
S = 4, correct target rays
S = 4, shuffled target rays
S = 4, zero target rays

Discussion

Number of source views. Quality rises with every additional source view, although the model was trained with exactly two: validation PSNR 15.17 → 16.11 → 16.65 → 16.83 dB for S = 1, 2, 3, 4 (test 15.50 → 16.16 → 16.98 dB for S = 1, 2, 4). Nothing in the architecture fixes S: attention runs over however many source tokens there are, and the positional embedding is per patch, not per view, so extra sources simply add observed content. A target pixel that is not seen in any source has to be hallucinated, and with one source a large part of the target is unobserved; each further source covers more of it. The gain shrinks (+0.9 dB from 1 to 2, +0.2 dB from 3 to 4) because the views come from one 32-frame window and overlap heavily. S = 6 (16.65 dB) is not higher than S = 3: it leaves only two targets per scene, so it averages over a different and smaller set of images, and six simultaneous sources are likely further from what the two-source training prepared the model for.

Ray conditioning. The target rays are what tells the model which view to render. Shuffling them (rays of another target view) drops PSNR by 4.7 dB at S = 2 and zero rays by 3.7 dB, consistently for S = 1, 2, 4 and on the test split. Wrong rays are worse than no rays: with shuffled rays the model confidently renders a different viewpoint whose structure is misplaced (lowest SSIM, 0.469), whereas with zero rays it falls back to a blurry average of the sources (SSIM 0.522) that is wrong in detail but right in overall colour and layout. Even the correct-ray result keeps only coarse structure here: the predictions are blurry and show 8×8 patch blocks, and training PSNR tracks validation PSNR and is still rising slowly at the end, so after 5600 steps the model is under-trained rather than over-fitted (see the longer run below).

2.2 (continued) The same model trained 4× longer

Because the 100-epoch model is clearly under-trained, I also trained it for 400 epochs (22,400 steps; everything else identical, the cosine schedule stretched to 400 epochs; 2.1 hours on one RTX PRO 5000). The 100-epoch run above is the configs/lvsm.json deliverable; this run checks whether the conclusions hold for a better-trained model. Best checkpoint: epoch 396 (validation PSNR 17.25 dB).

400-epoch training curves (same settings, cosine schedule over 400 epochs)

Validation, PSNR [dB] / SSIM

Scorrect raysshuffled rayszero rays
115.90 / 0.58911.62 / 0.48012.46 / 0.527
217.25 / 0.60011.18 / 0.45812.46 / 0.527
318.05 / 0.60911.19 / 0.44812.48 / 0.528
418.29 / 0.62011.25 / 0.45412.45 / 0.538
618.09 / 0.58610.97 / 0.42512.47 / 0.510

Test, PSNR [dB] / SSIM

Scorrect raysshuffled rayszero rays
116.30 / 0.57012.21 / 0.46512.71 / 0.510
217.31 / 0.58212.02 / 0.45312.72 / 0.512
318.16 / 0.59012.10 / 0.44712.78 / 0.513
418.50 / 0.58911.54 / 0.43112.71 / 0.508
619.18 / 0.60311.87 / 0.43912.85 / 0.524
100 epochs, S = 2
400 epochs, S = 2 (same scene and views)
400 epochs, S = 4

Four times the training raises validation PSNR at S = 2 from 16.11 to 17.25 dB (SSIM 0.557 → 0.600), and validation PSNR was still rising slowly at the end, so this model would keep improving with more compute. The predictions now place the coarse layout and the large objects (window, table, armchair) correctly, but fine texture is still blurred. Both conclusions above get stronger: the benefit of more sources grows (S = 1 → 4: +2.4 dB vs +1.7 dB at 100 epochs; test S = 6 reaches 19.18 dB), and so does the dependence on correct rays: shuffled rays now lose 6.1 dB at S = 2 and 7.1 dB at S = 6, while zero rays stay at ≈12.5 dB whatever S is — without a target pose, extra sources cannot help because the model does not know which view to render.

2.3 Bonus: does the number of training source views matter?

Question. The baseline always trains with S = 2 sources, yet it is evaluated with 1 to 6. Does training with a random number of sources per step, S ~ U{1, …, 4} (configs/lvsm_randS.json), make the model better at the source counts it would otherwise never see? Controlled comparison: identical model, data, optimiser, schedule, number of steps and seed; only the per-step S differs (with V = 8 views, the number of targets is 8 − S). best.pt is selected the same way for both (validation PSNR at S = 2) and both are evaluated with correct rays on the same validation scenes and views for S = 1, 2, 3, 4, 6. Because the 100-epoch models turned out to be under-trained, the comparison was repeated with 400 epochs for both models (declared before the 400-epoch runs: a difference counts only if it reaches 0.5 dB at some S, about five times the largest 100-epoch gap).

S at evaluationbaseline, 100 eprandom S, 100 epbaseline, 400 eprandom S, 400 ep
115.1715.1815.9015.94
216.1116.1117.2517.33
316.6516.6818.0518.25
416.8316.9118.2918.57
616.6516.7418.0918.63
validation PSNR/SSIM against the evaluation source count, 100 epochs
the same comparison after 400 epochs

Results and discussion

100 epochs: no effect. The two models agree to within 0.09 dB at every evaluation S. Both are still far from fitting the training data (training PSNR ≈ validation PSNR ≈ 16 dB), so what limits them is how much has been learned at all, not how the number of sources was distributed during training.

400 epochs: random S helps exactly where it should. With four times the training, the random-S model matches the baseline at S = 1 and 2 (+0.04 and +0.08 dB) but is better with more sources, and the gain grows with S: +0.20, +0.28 and +0.53 dB on validation for S = 3, 4, 6, and +0.17, +0.26, +0.48 dB on the test split. By the rule declared before these runs (≥ 0.5 dB at some S) the effect passes on validation and is just short on test; the consistent ordering on both splits is the stronger evidence. With shuffled or zero target rays the two models are identical (within ±0.06 dB), so the gain comes from better use of the sources, not from a different rendering prior.

Why. A baseline transformer only ever sees sequences with two source images, so attention learns to combine two views; with more sources at test time it still benefits (S = 2 → 4: +1.0 dB), but it has never learned how to weigh four or six partially redundant views against each other. The random-S model practises fusing one to four sources, and that skill also carries over to six, which neither model saw. At S = 1 nothing changes: the limit there is the content that the single source does not show and has to be hallucinated, which more varied training does not solve. Random S costs nothing measurable at S = 2 even though it spends fewer updates on two sources and, when S is large, supervises fewer targets per step (T = 8 − S).

Limitations. One training seed per model; at 100 epochs the two models differed by at most 0.09 dB, which is a rough estimate of run-to-run noise, so the S = 3 gain (+0.2 dB) is suggestive and the S = 4–6 gains (+0.3 to +0.5 dB) are clearer. A second seed per model would firm this up. S = 6 has only two targets per scene, so its average is over fewer images than the other columns.

3. Multi-View Geometry with Mini-VGGT

Implementation (Q3.0). aggregate alternates the 16 blocks: even blocks attend within each view (tokens reshaped to (B·V, N, D) with the per-view 2D RoPE positions), odd blocks attend across all views ((B, V·N, D), positions repeated per view, no view-order embedding since camera 0 is marked by its own camera/register tokens); the output of the last frame block is kept and concatenated with the last global block's output and normalised to 2D channels. CameraHead.predict applies the MLP in float32 and a ReLU on the two FoV channels only. DepthHead unpatchifies the linear patch predictions, adds the convolutional refinement of [depth, RGB] as a residual, and returns exp(depth) and 1 + exp(confidence) (both computed in float32). The losses follow the docstrings: element-wise L1 on the w2c translation (clamped at 100), quaternion and FoV (×0.5); trimmed-mean absolute depth error plus the confidence-weighted term c·e − 0.2 log c; and the 4-scale masked gradient loss on depth residuals (a CPU test checks it against a hand computation and that it keeps a gradient when no edge is valid).

3.1 Overfitting

overfit loss curve (total, camera and depth loss) from train.py

One fixed training clip (scene 036bce3393, 8 views), 40 epochs × 25 steps at learning rate 1e-3 (the --overfit defaults), evaluated on the same clip after every epoch. The total loss falls from 5.48 to -0.15 (it becomes negative because of the −0.2 log c confidence term once the depth error is small), the camera loss from 0.342 to 0.003 and the depth loss from 3.77 to -0.17. In metric terms, depth AbsRel goes from 0.619 to 0.039 (δ1 0.952) and the pairwise pose AUC@30 from 0.000 to 0.995 (rotation 1.000, translation 0.995): the predicted cameras and point cloud match the ground truth below almost exactly, so the model, heads and losses can fit the targets.

fixed training clip: ground-truth cameras and depth points
same clip: predicted cameras and depth (after overfitting)

3.2 Training and evaluation

Settings (configs/vggt.json defaults, num_views = 8, sampling_range = 32): frozen DINOv2 ViT-B/14 encoder; 384-wide aggregator with 8 frame/global pairs (16 blocks, 6 heads) and a 4-block camera transformer, 57.7 M trainable parameters; 224 px, batch 16 clips of 8 views (14 steps per epoch over the 224 training scenes); AdamW, learning rate 6e-4 with 10 warm-up epochs and cosine decay to 5% over 1000 epochs, weight decay 0.01, gradient clipping 1.0, loss weights camera 5 / depth 1, bfloat16 autocast; one A100. Training was stopped by my 5-hour budget after epoch 882 of the 1000-epoch schedule (learning rate then 5.0e-05, 8% of the peak, so the remaining epochs would have changed little); best.pt is epoch 821, the lowest validation loss among the saved epochs (0.860). The validation loss was still decreasing slowly at the end.

loss curve from train.py (total, camera and depth loss)
validation depth metrics and pose AUC@30 after every epoch (from history.jsonl)

Evaluation of best.pt with 8 views from the first 32 prepared frames, 224 px, seed 825, full splits:

splitAbsRel ↓δ1 ↑Rot AUC@5 ↑Rot AUC@15 ↑Rot AUC@30 ↑Trans AUC@5 ↑Trans AUC@15 ↑Trans AUC@30 ↑Joint AUC@5 ↑Joint AUC@15 ↑Joint AUC@30 ↑
val (28 scenes)0.14680.82600.18670.49280.67840.01300.08010.17830.00410.06740.1648
test (28 scenes)0.15060.81300.20030.53190.72410.01300.06120.17300.00310.04370.1575

Ground truth vs. prediction on four validation scenes

validation scene 1: ground truth
validation scene 1: prediction
validation scene 2: ground truth
validation scene 2: prediction
validation scene 3: ground truth
validation scene 3: prediction
validation scene 4: ground truth
validation scene 4: prediction

Discussion of held-out errors

Depth is the strongest output. AbsRel 0.147 / 0.151 and δ1 0.826 / 0.813 (val / test): about 82% of valid pixels are within 25% of the true depth, in units normalised by each clip's scene scale, which the model must also infer. The frozen DINO features carry strong single-image cues (layout, surface orientation, object boundaries). The screenshots show where the remaining error is: the predicted clouds reproduce walls, floor and large furniture, but thin structures such as chair legs and backs are smoothed into the background, and surfaces come out thicker and noisier than in the ground truth, because each view's depth and camera are predicted separately and their small scale or pose errors do not cancel when the views are fused.

Rotation is good, translation direction is not. Rotation AUC@30 is 0.678 / 0.724, but translation AUC@30 only 0.178 / 0.173, so the joint pose AUC (0.165 / 0.157) is limited by translation. All views come from a window of 32 consecutive frames of the provided sequences (every 10th iPhone frame), so baselines are short compared with the scene depth: a rotation changes where every pixel moves, while the direction of a few-centimetre translation is only visible in small parallax, and the L1 loss on the (small) translation vector barely penalises a wrong direction. The model also makes a single camera prediction, without VGGT's iterative refinement. Validation and test agree closely, so these are properties of the model and data rather than of particular scenes.

No over-fitting. The validation loss stays below the training loss and keeps decreasing; the training loss is an average over the epoch with random 8-view windows from anywhere in the training sequences, so it is not a measure of fit to fixed samples. Averaged over 50-epoch windows, validation AbsRel was still improving at the end (0.167 for epochs 700–749, 0.157 for 832–881), whereas rotation AUC@30 was nearly flat (0.669 → 0.679) and translation AUC@30 had levelled off at 0.17–0.18, which matches the explanation above: more of the same short-baseline training mainly improves depth, not the camera translation.

3.3 Bonus: how does the number of input views affect accuracy, runtime and memory?

Question. The model was trained with exactly 8 views. How do accuracy, runtime and memory change when the same trained model is given V = 2 … 32 views at test time? Evaluation only, one checkpoint (best.pt), same prepared splits, first-32-frame prefix and seed as the official evaluation, batch 1 for timing (forward pass only, A100, after a warm-up scene) — extras/vggt_views.py.

Making it a controlled comparison. The all-view metrics in (a) are averaged over different things for different V: at V = 2 half of the depth maps belong to the reference view and each scene contributes one pose pair, at V = 32 there are 496 pairs per scene. The prepared loader picks frame 0 plus a seeded random permutation of the other 31 frames, so the view sets are nested (V = 2 ⊂ 4 ⊂ … ⊂ 32, asserted in the script). Table (b) therefore scores only what every V shares — the depth of views 0 and 1 and the relative pose of the pair (0, 1) — while the model sees V views (extras/vggt_views_nested.py, 56 val+test scenes). Depth targets are normalised by a scene scale computed from all V views, so (b) also reports AbsRel after aligning each scene's two depth maps to the ground truth by their median ratio, and the mean |log| of that ratio, to separate scale inference from depth shape.

(a) All-view metrics (the standard evaluation, with V views)

Validation:

views VAbsRelδ1Rot AUC@30Trans AUC@30Joint AUC@30forward [ms/scene]peak memory [GiB]
20.11160.88410.7250.2440.24050.40.69
40.12490.85270.6740.1530.14353.30.71
80.14680.82600.6780.1780.16558.90.75
160.17570.80690.6360.1650.14067.20.83
240.16790.81110.6550.1650.14476.20.90
320.17410.80190.6550.1600.14065.10.98

Test:

views VAbsRelδ1Rot AUC@30Trans AUC@30Joint AUC@30forward [ms/scene]peak memory [GiB]
20.11920.85640.7580.1290.10448.30.69
40.13670.83150.7350.1720.15153.10.71
80.15060.81300.7240.1730.15859.90.75
160.15940.79880.7260.1720.15467.60.83
240.16060.79800.7290.1690.15278.60.90
320.16710.79380.7320.1700.15483.40.98
validation, standard all-view metrics and cost against V

(b) Controlled: the same two views for every V (validation + test, 56 scenes)

VAbsRel, views 0–1AbsRel after median scale alignmentmean |log scale ratio|δ1pair (0,1) rotation error, median [°]rotation AUC@30translation-direction error, median [°]translation AUC@30
20.11540.08490.0770.87025.710.74035.20.186
40.12410.08400.0780.84946.220.72637.00.176
80.14100.08350.0920.83146.370.71737.70.164
160.15730.08350.1010.81656.850.70734.90.185
320.15990.08350.1050.81016.950.69433.80.172
controlled comparison on views 0 and 1

Results and discussion

Cost. Runtime and memory grow slowly: 50 ms and 0.69 GiB per scene at V = 2, 59 ms / 0.75 GiB at the training setting V = 8, and 65 ms / 0.98 GiB at V = 32 (batch 1, A100; the timings are single measurements and noisy — V = 24 measured slower than V = 32 — so only the trend is meaningful). At these sizes the fixed cost dominates (16 aggregator blocks, the camera transformer and the DINO encoder launched once per scene); per-frame work is small and global attention over V × 261 tokens is still short (8,352 tokens at V = 32), so the quadratic term has not taken over yet.

Accuracy: fewer views are better for this model, and the controlled table shows why. The standard all-view metrics in (a) get worse with V (validation AbsRel 0.112 → 0.147 → 0.174), but they mix in a changing population of views and pairs. On the two views shared by every V (b), the same trend remains (AbsRel 0.115 → 0.141 → 0.160; rotation error of the pair (0, 1) 5.7° → 6.4° → 6.9°), so it is real, not an averaging artefact. After aligning each scene's depth scale, however, AbsRel is flat (0.0849, 0.0835, 0.0835), while the scale error grows (mean |log ratio| 0.077 → 0.092 → 0.105). The shape of the predicted depth does not depend on V at all, which is consistent with the depth shape coming largely from per-image features, and the whole depth degradation is the global scale.

Why the scale. Targets are normalised so that the centroid of the valid points of all V views lies at distance 1 from camera 0. With V = 2 that scale is a property of the two images the network sees in full; with V = 32 it depends on 30 more views spread over the 32-frame window, and the network has to estimate a quantity it was only trained to estimate for 8 views. Errors in that estimate scale every depth map at once. The pose of a fixed pair degrades more mildly (+1.2° median from V = 2 to 32), as global attention spreads over more frames than in training; translation direction stays poor at every V (median 34–38°, see Q3.2).

Limitations and what would change it. One trained model (V = 8), evaluation only; training with a range of V (as in Q2.3 for Mini-LVSM) or making the scale normalisation independent of V (e.g. reference-view-only scale) would be the natural next experiments.

AI assistance

Anthropic Claude was used to explain course concepts and to polish the writing.