Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision

Name: Alicia Yang  |  Andrew ID: aliciay

Contents

  1. Multi-View Data Preprocessing: visualizations · final check · canonicalization, rays and Plücker coordinates
  2. Novel-View Synthesis with Mini-LVSM: 2.1 · 2.2 · 2.3 (bonus)
  3. Multi-View Geometry with Mini-VGGT: 3.1 · 3.2 · 3.3 (bonus)

1. Multi-View Data Preprocessing

All Part 1 code is in a4data/geometry.py. I follow the conventions of the data contract. Cameras use OpenCV axes (+X right, +Y down, +Z forward). Poses are camera-to-world: Xworld = R Xcam + t.

1.2–1.3 Data Visualizations

Both pages come from scripts/visualize_data.py --num-scenes 5 --max-points 10000. They show the same five training clips (8 views each, seed 825). The left images are after unprojection. They are in the original ScanNet++ world frame, in meters. The right images are after canonicalization. They are in camera 0's frame (the red frustum). The side panel shows view 0's RGB and depth, the first camera center, and the distance to the centroid of the valid points. Click an image to see it at full size.

ClipAfter unprojection (m)After canonicalization
First camera centerCentroid distanceFirst camera centerScene scale sCentroid distance
036bce3393
frame 4470
(3.3463, 4.6765, 1.4680)6.668(0, 0, 0)1.867 m1.000
076c822ecc
frame 2890
(1.5398, 2.7673, 1.5862)4.435(0, 0, 0)2.830 m1.000
079a326597
frame 7430
(3.7559, 2.0824, 1.5325)2.718(0, 0, 0)2.080 m1.000
07f5b601ee
frame 3630
(3.0944, 3.4889, 1.4076)5.118(0, 0, 0)0.924 m1.000
07ff1c45bb
frame 460
(1.3404, 2.1707, 1.5162)3.285(0, 0, 0)1.966 m1.000

The numbers are as the viewer prints them. Before canonicalization, the centroid distance is only the distance from ScanNet++'s arbitrary world origin. After canonicalization, camera 0 is at the origin (the viewer prints ±0.0000). The centroid is 1.000 units away in every clip. The scale that was divided out ranges from 0.92 to 2.83 m. In both stages, the eight views fit together into one consistent room.

Unprojection viewer, clip 036bce3393
036bce3393, after unprojection
Canonicalization viewer, clip 036bce3393
036bce3393, after canonicalization
Unprojection viewer, clip 076c822ecc
076c822ecc, after unprojection
Canonicalization viewer, clip 076c822ecc
076c822ecc, after canonicalization
Unprojection viewer, clip 079a326597
079a326597, after unprojection
Canonicalization viewer, clip 079a326597
079a326597, after canonicalization
Unprojection viewer, clip 07f5b601ee
07f5b601ee, after unprojection
Canonicalization viewer, clip 07f5b601ee
07f5b601ee, after canonicalization
Unprojection viewer, clip 07ff1c45bb
07ff1c45bb, after unprojection
Canonicalization viewer, clip 07ff1c45bb
07ff1c45bb, after canonicalization

1.7 Final Check

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 on the full training split (224 clips):

SampleViewsResolutionValid depthMax point reprojection errorMax ray reprojection errorCentroid distance
e050c15a8d
frame 7080
822497.8%1.13e-4 px1.09e-4 px1.0
5f99900f09
frame 2820
822499.0%1.23e-4 px1.11e-4 px0.99999994
47b37eb6f9
frame 2720
822496.1%9.88e-5 px8.90e-5 px1.0
4c5c60fa76
frame 6690
822475.2%1.09e-4 px1.03e-4 px0.99999994

Batch shapes (batch size 2 from data.json, V = 8 views, 224×224):

KeyShapeKeyShape
rgb(2, 8, 3, 224, 224)points_world(2, 8, 3, 224, 224)
depth(2, 8, 1, 224, 224)ray_origins(2, 8, 3, 224, 224)
depth_valid(2, 8, 1, 224, 224)ray_directions(2, 8, 3, 224, 224)
K(2, 8, 3, 3)plucker(2, 8, 6, 224, 224)
T_world_from_camera(2, 8, 4, 4)frame_numbers(2, 8)
original_world_from_reference(2, 4, 4)scene_scale(2,)
Every reprojection error is about 1e-4 px. That is far below the 0.01 px limit. Every centroid distance is 1, up to float32 rounding. The only detail that mattered for that limit was in resize_intrinsics. Pixel centres are at integers, so a resize by s maps c → s·c + (s − 1)/2, not s·c. For the 300 px crop resized to 224, leaving out that offset would shift every pixel by about 0.13 px.

1.3–1.5 Canonicalization, Ray Directions and the Plücker Moment

Canonicalization Each training clip is moved into the frame of its first sampled camera and then rescaled. Let T0 be camera 0's camera-to-world pose. Every pose becomes Ti' = T0−1 Ti, so camera 0's pose is the identity. Then I divide translations, depths and points by s. Here s is the distance from camera 0 to the centroid of the valid points. After this, the centroid is 1 unit from the origin. Rotations do not change. I do not subtract the centroid, because camera 0 has to stay at the origin.

Why: each ScanNet++ scene has its own arbitrary world frame, and rooms differ in size. From images alone, you cannot work out a global rigid transform or a global scale. Canonicalization removes both, so a network never has to guess them. The evaluation loader does the same thing.

World-space ray directions Pixel p = (u, v, 1) looks along K−1p in the camera frame. To get the world direction, I rotate that by the camera-to-world rotation and normalize it:
d = R K−1p / ‖R K−1p‖,    o = t
Only R acts on directions, because translation does not change a direction. Every ray of a camera starts at the camera center t. The poses are canonical, so the rays are canonical too. Depth is z-depth, not distance along the ray. So the point at depth z is o + (z / dz)·d, where dz is the z component of d in the camera frame. This gives back the unprojected points. The ray reprojection error above checks each direction. I rotate it back into the camera and project it through K, and it should land on its own pixel.

The Plücker moment Writing a ray as (o, d) is redundant, because any point on the line could be o. Plücker coordinates [d, m] with m = o × d remove that extra freedom. If I slide the origin to o + λd, I get (o + λd) × d = o × d, because d × d = 0. So m depends only on the line. It is perpendicular to d. When d is a unit vector, the length of m is the distance from the origin to the line. d says which way the ray points, and m says where the line is. So a network gets each pixel's ray as 6 channels with no ambiguity. Camera 0's rays pass through the origin, so their moments are 0. Rescaling by s scales the moments by 1/s, the same as the origins.

2. Novel-View Synthesis with Mini-LVSM

All Part 2 code is in models/lvsm/. Mini-LVSM (14.6M parameters) predicts a target image from S source images and the target camera's Plücker rays. It never sees the target's RGB. Each source patch has RGB (shifted to [−1, 1]) plus rays, so 9 channels. Each target query patch has only the 6 ray channels. patchify cuts 8×8 patches (a 28×28 grid, so N = 784 per view). Two linear layers embed them. The same 2D sin-cos position embedding is added to every view. Each target gets its own sequence: all S·N source tokens, then its own N query tokens. So targets never see each other. After 8 full-attention blocks, I keep only the query tokens. I decode them with the RGB head and a sigmoid, then run unpatchify. The loss is MSE. I compute it in float32, outside the bf16 autocast.

2.1 Overfitting

python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit trains for 2000 steps on one fixed training batch (4 clips × 6 targets = 24 images). Then it scores that same batch.

Final PSNRFinal SSIMFinal MSEMean train PSNR over the 2000 steps
30.90 dB0.8700.0008725.37 dB
Overfit batch: two source views, ground truth and prediction for four targets
outputs/lvsm_overfit/val_epoch000.png: source view 1 | source view 2 | ground truth | prediction
What the figure shows save_visualization takes the first clip in the batch. It draws one row per target view (4 of its 6). Each row shows the same two source views, then the ground-truth target, then my prediction. The predictions match the ground truth closely. This is true even for the two rows that face the dark-blue wall, which neither source shows. This is memorization, not novel-view synthesis, because here the "validation" batch is the training batch. Faint 8×8 patch edges are still visible, because each query token decodes its own patch. The mean train PSNR is lower because it includes the early steps.

2.2 Training and Evaluation

Training settings

SettingValue
Configsconfigs/lvsm.json with configs/scannetpp/iphone/data.json
Training data224 training scenes; 8 views per sample from a 32-frame window; 224×224
ViewsS = 2 sources and T = 6 targets per sample
Batch and lengthB = 4, so 56 steps per epoch; 100 epochs = 5,600 steps
Modelpatch 8, dmodel 384, 8 layers, head dim 64, MLP ratio 4, QK-norm
OptimizerAdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), gradient clip 1.0
Schedule200 warmup steps, then cosine decay to 10% of the peak
Precisionbf16 autocast (AMP), loss in float32
Validation28 val scenes, 8 fixed views from the first 32 prepared frames, S = 2 (168 target images); best.pt = best val PSNR
Hardwareone NVIDIA L40S, about 29 minutes

Training curves

Train and validation loss, PSNR and validation SSIM over 100 epochs
From outputs/lvsm/log.jsonl. Train values are averaged over the steps of each epoch. Validation uses the full val split at S = 2.
Validation clip at epoch 95: two sources, ground truth and prediction
Training visualization val_epoch095.png, the last one saved (one is saved every 5 epochs): source 1 | source 2 | ground truth | prediction

Ablations: number of source views and ray conditioning

python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none on the val split. best.pt is from the last epoch (99). Shuffled gives each target the rays of a different target (shifted by one). None sets all target rays to zero. Every S uses the same 8 views per scene. Only the split into sources and targets changes.

STargets (images)CorrectShuffledNone
PSNRSSIMPSNRSSIMPSNRSSIM
17 per scene (196)15.19 dB0.55811.80 dB0.48712.46 dB0.529
2 (trained)6 per scene (168)16.14 dB0.56011.49 dB0.47212.44 dB0.528
44 per scene (112)16.86 dB0.57511.59 dB0.47312.46 dB0.538

These numbers are within about 0.2 dB of the course staff's reference solution with the same config (Piazza @89: about 15, 16 and 17 dB PSNR and 0.56, 0.56 and 0.57 SSIM at S = 1, 2, 4).

Evaluation images for the first val clip (source views | ground truth | prediction):

S = 1, correct rays
S = 1, correct rays
S = 2, correct rays
S = 2, correct rays
S = 4, correct rays
S = 4, correct rays
S = 2, shuffled rays
S = 2, shuffled rays
S = 2, no rays
S = 2, no rays
S = 1, shuffled rays
S = 1, shuffled
S = 1, no rays
S = 1, none
S = 4, shuffled rays
S = 4, shuffled
S = 4, no rays
S = 4, none

Discussion

Number of source views More source views help. PSNR goes 15.19 → 16.14 → 16.86 dB for S = 1, 2, 4, even though the model only trained with S = 2. Nothing in the architecture fixes S. Attention just sees a shorter or longer sequence. More sources cover more of each target, so the model has to guess less. SSIM goes up the most at S = 4. Note that the target images are not the same across S (7, 6 and 4 per scene), so the comparison is not perfectly controlled.

Ray conditioning The target rays are the only thing that tells the model where the target camera is. They matter a lot. At S = 2, correct rays give 16.14 dB, no rays give 12.44 dB and shuffled rays give 11.49 dB. Without the right rays, more sources do not help at all (the none and shuffled columns stay flat across S). In the images, no rays gives an almost flat beige image. Every query token is the same except for its position, so the model falls back to the scene's average colour. Shuffled rays give a blurry rendering from another target's camera.

Why shuffled rays are worse than none MSE rewards safe, averaged guesses. With no pose, the best the model can do is the average colour. That is the best constant guess under MSE. With a wrong pose, the model commits to a layout (dark chairs, a bright window) but puts it in the wrong places. So it is penalized for the structure it adds and also for the structure it misses. If the wrong view had nothing in common with the target, the error would be about double the error of the flat guess (3 dB). The actual gap is 0.7–1.0 dB. I think this is because the shuffled renders are blurry and neighbouring views overlap. SSIM shows the same thing (0.472 vs 0.528 at S = 2), because its structure term penalizes wrong edges.

Blurry results and undertraining The predictions are blurry and blocky, at about 16 dB at S = 2. They get the rough layout and colours (window, table, dark chair) but no detail. The 8×8 patch grid is visible. I think the main reason is the short default schedule. 5,600 steps was not enough. Validation PSNR was still rising at the end (15.95, 16.03 and 16.06 dB averaged over epochs 70–79, 80–89 and 90–99). The best checkpoint is also the last epoch. Train and validation stay close the whole time (15.60 vs 16.14 dB at the end), so the model is underfitting, not overfitting. That is why I tried a 4× longer schedule, below.

Training longer: 400 epochs

The Piazza answer above confirms that the model is underfit. It also allows longer training if the settings are stated. configs/lvsm_long.json is configs/lvsm.json with 400 epochs instead of 100 (22,400 steps), and it saves validation images every 20 epochs. The cosine schedule now spans all 400 epochs. Data, model, optimizer, peak learning rate, warmup and seed are unchanged. It took 97 minutes on one L40S. That run had 8 CPUs and the 100-epoch run had 4, so the wall times are not directly comparable. best.pt is epoch 378. I evaluated it with the same command as above plus --lpips (AlexNet LPIPS, lower is better). For the 100-epoch LPIPS values, I reran that model's evaluation with --lpips.

S100 epochs (above)400 epochsΔ PSNR
PSNRSSIMLPIPS ↓PSNRSSIMLPIPS ↓
115.19 dB0.5580.70515.91 dB0.5880.659+0.72 dB
2 (trained)16.14 dB0.5600.67317.20 dB0.5990.607+1.07 dB
416.86 dB0.5750.65218.39 dB0.6210.583+1.53 dB

Correct rays. With shuffled target rays the 400-epoch model gets 11.2–11.6 dB, and with no target rays it gets 12.4–12.5 dB. This is the same as the 100-epoch model (outputs/lvsm_long/eval_val/metrics.json).

Validation PSNR, validation SSIM and train PSNR per epoch for the 100- and 400-epoch runs
From both runs' log.jsonl. Validation is at S = 2.
Ground truth, 100-epoch and 400-epoch predictions at S = 2
S = 2: ground truth | 100 epochs | 400 epochs
Ground truth, 100-epoch and 400-epoch predictions at S = 4
S = 4: ground truth | 100 epochs | 400 epochs
What changed Every metric improves at every S. The gain grows with the number of source views: +0.72, +1.07 and +1.53 dB at S = 1, 2, 4. In the images, the 400-epoch model draws chairs, table legs and window frames that you can recognise. The 100-epoch model had colour blobs in those places. Fine texture is still missing, and the 8×8 patch grid is still visible.

Where the gain comes from With shuffled rays or no rays, the longer model is no better than the short one. So the extra quality comes from making better use of the target pose, not from a better average guess.

No more gain at this model size Validation PSNR averaged 17.17, 17.16 and 17.17 dB over the last three 10-epoch windows, while the learning rate decayed to 10% of its peak. So the model has stopped improving under this schedule. Train PSNR (16.60 dB at the end, on random clips) is still below validation, so the model is still not overfitting. The next step would be more capacity, such as a wider or deeper model, or smaller patches. The Piazza answer lists these as bonus directions.

2.3 Bonus: Training View Count, and a Perceptual Loss

Question The baseline always trains with S = 2. Does training with a random number of source views per step, S ∈ {1, 2, 3, 4}, change how well the model does when it is evaluated with 1, 2 or 4 source views?

Settings

configs/lvsm_random_views.json is configs/lvsm.json plus "num_source_views_range": [1, 4]. Each training step draws S uniformly from 1 to 4 for the whole batch. The other 8 − S views are targets. Everything else is the same: data, model, optimizer, schedule, 100 epochs and seed. Validation during training stays at S = 2, so the two curves can be compared directly. After training, I evaluated both best.pt files (epoch 99 in both runs) with --num-source-views 1 2 4 and correct rays.

Results

Eval SFixed S = 2 (baseline)Random S ∈ [1, 4]Δ PSNR
PSNRSSIMPSNRSSIM
115.19 dB0.55815.27 dB0.559+0.08 dB
216.14 dB0.56016.22 dB0.560+0.09 dB
416.86 dB0.57517.10 dB0.575+0.25 dB

Δ is computed before rounding. Wall time was 28.9 vs 29.2 min, and peak GPU memory was 11,050 vs 12,256 MiB (baseline vs random S, from the job log).

Validation loss and PSNR curves of both runs, and evaluation PSNR versus number of source views
Left and middle: validation curves of both runs (log.jsonl, S = 2). Right: evaluation PSNR vs S (eval_val/metrics.json, correct rays).
Baseline evaluated at S = 4
Fixed S = 2, evaluated at S = 4
Random-S model evaluated at S = 4
Random S, evaluated at S = 4
Random-S model evaluated at S = 1
Random S, evaluated at S = 1
Random-S model evaluated at S = 2
Random S, evaluated at S = 2

The two models are hard to tell apart by eye. Both are as blurry as in 2.2.

Conclusions

What I found Training with random S was slightly better at every S I evaluated. The biggest gain was at S = 4 (+0.25 dB), the setting furthest from how the baseline was trained. At S = 2 it was also ahead on the validation curve in each of the last 20 epochs, by 0.11 dB on average. SSIM did not change (within 0.001).

Why it might help The baseline never saw more than 2 source views. So at S = 4, its attention gets a longer sequence than it trained on. For the random-S model, S = 1 and S = 4 are part of training. It only spent about a quarter of its steps at S = 2, and it still matched the baseline there. My guess is that mixing S works a bit like data augmentation, because each scene is seen with different amounts of context. I did not test that.

Limitations
  • One seed per setting, and small gains (+0.08 / +0.09 / +0.25 dB). So I cannot rule out run-to-run noise. A steady lead over 20 epochs shows that the two runs differ. It does not show why.
  • Train PSNR is not comparable. The random-S run's training PSNR mixes S = 1–4. So I only compare validation curves and evaluations.
  • Both models are undertrained (see 2.2). The ranking could change with a longer schedule.
  • Cost. Steps with S = 3 or 4 have longer sequences, so peak memory went up by about 1.2 GB. The wall time barely changed (29.2 vs 28.9 min). This is because the data loader is the bottleneck. In both runs the GPU was only about 60% busy, while the instance's 4 CPUs were 82% busy.

Second exploration: a perceptual loss

Question The predictions are blurry. MSE rewards blur, because when the model is unsure, an average is its best guess. Does adding a perceptual loss (LPIPS) make the predictions sharper? What does it cost in PSNR and SSIM?

configs/lvsm_lpips.json is configs/lvsm.json plus "loss": {"lpips_weight": 0.05, "lpips_net": "vgg"}. So the training loss is MSE + 0.05 · LPIPSVGG over every target image (MSEPlusLPIPS in models/lvsm/losses.py). The LPIPS network is pretrained and frozen. It runs in float32, outside the bf16 autocast. Its input images are mapped from [0, 1] to [−1, 1]. The weight 0.05 is the one that pixelSplat and MVSplat use. At the start of training, the weighted LPIPS term was 0.035 and the MSE was 0.065, so the two terms start at a similar size. Data, model, schedule, 100 epochs and seed are the same as the baseline. The validation loss is still plain MSE. best.pt is still picked by validation PSNR (epoch 99, the last epoch). For evaluation I report LPIPS with the AlexNet backbone, which is the usual metric. This way the model is not scored by the same network it trained against. Wall time was 35 min on the 8-CPU instance. Peak GPU memory was 15,136 MiB, against 11,050 MiB for the baseline.

Eval SMSE (baseline)MSE + 0.05 LPIPSΔ PSNR
PSNRSSIMLPIPS ↓PSNRSSIMLPIPS ↓
115.19 dB0.5580.70514.99 dB0.5150.711−0.20 dB
2 (trained)16.14 dB0.5600.67315.79 dB0.5120.689−0.34 dB
416.86 dB0.5750.65216.29 dB0.5190.682−0.56 dB

Correct rays, full val split. The LPIPS run's best.pt and last.pt give identical numbers because both are epoch 99.

Validation PSNR and SSIM of both runs, and the MSE and weighted LPIPS training terms of the LPIPS run
Left and middle: validation curves of both runs at S = 2. Right: the two training loss terms of the LPIPS run, averaged per epoch.
Ground truth, MSE and MSE plus LPIPS predictions at S = 2
S = 2: ground truth | MSE | MSE + LPIPS
The same 64 by 64 crop from the MSE and the MSE plus LPIPS predictions, enlarged four times
The same 64×64 crop (a window in the third row), enlarged 4×: MSE | MSE + LPIPS

Conclusions (perceptual loss)

What I found The perceptual loss made every metric worse at every S. This includes LPIPS itself (0.673 → 0.689 at S = 2). The PSNR drop grows with S: −0.20, −0.34 and −0.56 dB. The images are not sharper either. If anything, object outlines are blurrier. Every 8×8 patch also gets a fine, regular dot pattern (right image) that is not in the scene.

Why I think it happens At about 16 dB, the model cannot predict where the real edges are. My guess is that it can still lower the VGG distance without knowing that. It does this by adding generic high-frequency texture inside each patch. That texture is in the wrong place, so MSE and SSIM get worse. AlexNet is a different network, and it does not reward the texture either. The curves fit this. The weighted LPIPS term only fell from 0.035 to 0.028 over training. Even so, validation PSNR and SSIM stayed below the baseline for the whole run. The run's final train MSE was also higher (0.0357 vs 0.0342).

Limitations
  • One weight and one seed. A smaller weight might behave differently. So might training with MSE first and adding LPIPS later. I did not test either.
  • A short schedule. Both runs are 100 epochs long. At that length the model is still undertrained (see the 400-epoch run in 2.2). A perceptual loss may only help once the model can put edges in the right place.
  • The patch decoder may matter. Each 8×8 patch is decoded on its own. That could make it cheap to add the same texture everywhere. I did not test this.

3. Multi-View Geometry with Mini-VGGT

All Part 3 code is in models/vggt/. Mini-VGGT takes V RGB images and predicts every camera and a depth map for each view, in one forward pass. The point clouds come from unprojecting the predicted depth with the predicted cameras.

Tokens A frozen DINOv2 ViT-B/14 turns each 224×224 view into a 16×16 grid of 768-d features. These are projected to 384. Each view also gets one camera token and four register tokens. Camera 0 has its own learned set of these tokens. That is the only thing that tells the network which image defines the world frame.

Alternating attention (aggregate) The 16 blocks alternate between two types. Frame blocks attend within one image, with shape (B·V, N, D). They build context within each image. Global blocks attend across all views, with shape (B, V·N, D). This is where cross-view matching, relative pose and a shared scale come from. The heads take LayerNorm(concat(last frame output, last global output)). So I save the frame output before the last global block overwrites it.

Heads The camera head runs 4 more blocks over the camera tokens. Then an MLP outputs translation (3), a quaternion (4) and the field of view (2, through a ReLU). Its last layer is initialized to give the identity rotation, zero translation and a 53° FoV. The depth head maps each patch token to 14×14 values. Then a small conv net on [coarse depth, RGB] adds a residual that starts at zero. This residual lines up depth edges with image edges. Depth = exp(·) and confidence c = 1 + exp(·) ≥ 1.

Losses The camera loss (weight 5) is L1 on translation, quaternion and 0.5 × FoV against the GT pose encoding. The depth loss (weight 1) has three parts: a trimmed L1 term, a confidence-weighted term c|e| − 0.2 log c, and a gradient loss on the residual at strides 1, 2, 4 and 8. The gradient loss rewards correct local shape. The confidence term is a Laplace likelihood. The network learns to give high confidence where it expects small errors.

3.1 Overfitting

python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit fits one fixed training clip for 40 epochs × 25 steps. The clip is 036bce3393_frame_004470, the first clip from Part 1, with 8 views. Training uses lr 1e-3, no warmup and cosine decay to 5%.

Overfit total, camera and depth loss over 40 epochs
outputs/vggt_overfit/loss_curve.png: total, camera and depth loss on the clip after each epoch (epoch 0 is before training)
Metric (on the clip)Before trainingAfter 40 epochs
Total loss5.48−0.169
Camera loss (unweighted)0.3420.0025
Depth loss (unweighted)3.77−0.181
Mean rotation error49.7°0.22°
Mean translation-direction error90.0°0.42°
Depth AbsRel0.6190.036
Depth δ10.2080.956
Rotation AUC @5 / 15 / 300 / 0.050 / 0.1421.000 / 1.000 / 1.000
Translation AUC @5 / 15 / 300 / 0 / 00.986 / 0.995 / 0.998
Joint pose AUC @5 / 15 / 300 / 0 / 00.986 / 0.995 / 0.998

From the overfit_train entries of history.jsonl (28 view pairs). The translation error starts at exactly 90°. This is because the camera head starts at zero translation, and the metric gives a zero baseline an error of 90°.

Why the loss goes negative The total loss is 5 × camera + depth. It crosses zero at epoch 25. This is expected. The depth loss contains c|e| − 0.2 log c, with c ≥ 1. Once the errors are small, the network raises its confidence, and −0.2 log c keeps going down. So the depth loss can go below 0. The metrics above are the real check.
Overfit clip, ground-truth geometry
Ground truth
Overfit clip, predicted cameras and depth
Predicted camera + depth
These are screenshots of outputs/vggt_overfit/visualization/index.html (default camera, view 0). The predicted cloud has the same shape, scale and camera layout as the GT. The model learned to put camera 0 at the origin. Its predicted center is (0.0002, −0.0000, 0.0002), and the centroid distance is 0.999. The predicted depth map is smoother than the GT and shows patch-sized blocks. The cloud also has more stray points along depth edges.

3.2 Training and Evaluation

Training settings

SettingValue
Configconfigs/vggt.json (defaults)
Training data224 training scenes; batch 16 clips × 8 views from a 32-frame window (sampling_range 32); 224×224; 14 steps per epoch
Modelfrozen DINOv2 ViT-B/14; 8 frame/global block pairs, width 384, 6 heads; camera head with 4 extra blocks; depth head with RGB refinement
OptimizerAdamW, lr 6e-4, weight decay 0.01, gradient clip 1.0
Schedule1000 epochs; 10 linear warmup epochs, then cosine decay to 5% of the peak
Loss weightscamera 5, depth 1
Precision and loadingbf16; 4 data workers (4 was faster than 8 in a short benchmark on this 4-CPU instance)
Hardwareone NVIDIA L40S; 5.6 h of training wall time, in two segments. Epochs 0–33 took 16 min with outputs on the EBS root disk. The run then resumed from last.pt for epochs 34–1000, which took 320 min (19.9 s per epoch), with outputs moved to the instance's local NVMe disk. Each checkpoint is about 1 GB, and rewriting last.pt every epoch at the root disk's 125 MB/s was adding 8–16 s per epoch. The resume continues the same 1000-epoch schedule; the final learning rate is 3.0e-5, 5% of the peak as planned. The TA's reference took about 4.4 h on one A100.
Evaluationbest.pt (lowest val loss); 8 views from the first 32 prepared frames, 224 px, seed 825; full val and test splits (28 scenes each)

Loss curve

Full-training total, camera and depth loss over 1000 epochs, train and validation
outputs/vggt_train/loss_curve.png: total, camera and depth loss per epoch, train and validation (epoch 0 is before training)
Val, after epochTotal lossCamera lossDepth lossAbsRelMean rotation errorMean translation errorPose AUC@30
0 (init)4.080.2063.050.72634.5°90.0°0.000
1001.200.1720.3400.17126.9°56.0°0.080
5000.9350.1270.2990.16115.3°45.0°0.118
888 (best.pt)0.7940.1100.2470.13511.5°40.7°0.183
1000 (last)0.8160.1120.2570.14111.6°41.0°0.175
The depth loss drops mostly in the first 100 epochs and then flattens. The camera loss keeps falling almost linearly until about epoch 900. So pose looked like the part where more training would help most. I tested that at the end of this section. The validation loss stays below the training loss for the whole run, so there is no sign of overfitting. But the two curves are not measured on the same kind of sample. Validation uses fixed clips (the first 32 prepared frames, seed 825). Training draws new random clips at every step.

Validation and test metrics

These are the metrics of best.pt (epoch 888), from outputs/vggt_train/metrics.json (val) and test_metrics.json (test). Each split has 28 scenes with 8 views each. That gives 28 × 28 = 784 ordered view pairs for the pose AUCs. There is no scale or shift alignment. So depth is compared directly in the normalized units of Part 1.

SplitAbsRel ↓δ1 ↑Rotation AUC ↑Translation AUC ↑Joint pose AUC ↑
@5@15@30@5@15@30@5@15@30
Validation0.1350.8510.1760.5070.7030.0170.0920.2080.0020.0630.183
Test0.1610.7790.1810.5250.7290.0170.0850.1830.0030.0580.164

Mean errors, val / test: rotation 11.5° / 9.4°, translation direction 40.7° / 42.4°, normalized depth RMSE 0.177 / 0.207.

GT vs. predicted geometry on four validation scenes

Screenshots of outputs/vggt_train/visualization/index.html, which the val evaluation of best.pt writes (default camera, view 0). Each row is one validation scene. The ground truth is on the left. The right shows the predicted cameras, with the predicted depth unprojected using those cameras.

Val scene 09c1414f1b, ground truth
09c1414f1b (dining room): ground truth
Val scene 09c1414f1b, predicted camera and depth
09c1414f1b: predicted camera + depth
Val scene 21d970d8de, ground truth
21d970d8de (classroom): ground truth
Val scene 21d970d8de, predicted camera and depth
21d970d8de: predicted camera + depth
Val scene 280b83fcf3, ground truth
280b83fcf3 (hallway): ground truth
Val scene 280b83fcf3, predicted camera and depth
280b83fcf3: predicted camera + depth
Val scene 2b1dc6d6a5, ground truth
2b1dc6d6a5 (office): ground truth
Val scene 2b1dc6d6a5, predicted camera and depth
2b1dc6d6a5: predicted camera + depth

Errors on held-out scenes

Depth: right layout, but too smooth On held-out scenes, AbsRel is 0.135 on val and 0.161 on test. On the overfit clip it was 0.036. δ1 is 0.85 and 0.78. So most pixels are within 25% of the GT, including scale, even with no alignment. The screenshots show where the rest of the error comes from: depth edges and thin structures (chair legs, desk edges, a chair's outline). The model works on a 16 × 16 grid of 14-pixel patches. The dense head upsamples from that grid. The small RGB refinement only partly brings back detail smaller than a patch. So boundaries bleed, and thin objects merge into the background.
Pose: rotation works, translation direction does not yet Rotation is reasonable. The mean error is 11.5° on val and 9.4° on test, and AUC@30 is 0.70–0.73. The translation direction is the weak part. Its mean error is about 41°, and its AUC@30 is about 0.2. The joint pose AUC uses the larger of the two errors, so translation limits it (0.18 / 0.16 at @30, and nearly 0 at @5). Two likely reasons:
  • The 8 views come from a 32-frame window of a handheld iPhone walk-through. So the baselines (the distances between cameras) are often small. With a small baseline, a few centimetres of error in the camera center is a large error in direction. The bunched cameras in the dining-room scene are an example.
  • The camera loss was still falling at epoch 1000. But 300 more epochs (below) barely changed the pose AUCs. So I don't think training time is the main limit.
Val vs. test Neither split is better on every metric. Test is worse on depth (AbsRel 0.161 vs 0.135) and translation, but better on rotation (9.4° vs 11.5°). Choosing best.pt by val loss favours val slightly. But each split has only 28 scenes. The error also depends a lot on the scene (room size, and how much of the view is wall or floor with no texture). So differences between scenes can explain these gaps.

Training longer: 300 more epochs

To test whether more training helps, I fine-tuned best.pt for 300 more epochs with configs/vggt_ft8.json. Data, model and loss weights are the same. The weights are loaded with --init-checkpoint, which loads only the weights and starts a fresh optimizer. Then there is a new 5-epoch warmup to lr 2e-4 (a third of the original peak) and cosine decay to 5%. It took 85 minutes (16.3 s per epoch) on one L40S, with 6 data workers on an 8-CPU instance. The fine-tune's best.pt is epoch 176. Everything else on this page still uses the original epoch-888 checkpoint.

Fine-tune total, camera and depth loss over 300 epochs, train and validation
outputs/vggt_ft8/loss_curve.png. Epoch 0 is the original best.pt before fine-tuning.
Split and modelAbsRel ↓δ1 ↑Rotation AUC ↑Translation AUC ↑Joint pose AUC ↑
@5@15@30@5@15@30@5@15@30
Val, original0.1350.8510.1760.5070.7030.0170.0920.2080.0020.0630.183
  +300 epochs0.1370.8580.2030.5180.7050.0170.0900.2110.0020.0600.182
Test, original0.1610.7790.1810.5250.7290.0170.0850.1830.0030.0580.164
  +300 epochs0.1580.7870.1790.5140.7170.0110.0680.1810.0030.0480.165
The model has plateaued Restarting at lr 2e-4 first made things worse. Validation loss rose from 0.794 to about 0.83 within 50 epochs. Then it came back down as the learning rate decayed, to a best of 0.779 at epoch 176 (0.805 at the end). The metrics of that checkpoint move in both directions. On val, δ1 and rotation AUC@5 improve (0.851 → 0.858, 0.176 → 0.203), but AbsRel gets slightly worse (0.135 → 0.137). On test, depth improves a little, while rotation AUC@15 and translation AUC@15 drop (0.525 → 0.514, 0.085 → 0.068). With 28 scenes per split, these changes are within scene-to-scene noise. So after 1000 epochs, this model has learned about as much as it can from this data at this size. More epochs mostly move the errors around. That is why 3.2 and 3.3 keep the original checkpoint.

3.3 Bonus: Number of Input Views

Question The model trains with V = 8 views. How does the number of input views at evaluation time (V = 2, 4, 8 or 16) affect accuracy, runtime and memory?

Settings

python scripts/vggt_num_views.py --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --eval-num-frames 32 --image-size 224 --seed 825 --views 2 4 8 16 --batch-size 1 --device cuda --output outputs/vggt_train/num_views

Results

From outputs/vggt_train/num_views/num_views.json (val, 28 scenes). The V = 8 "all views" row matches the Q3.2 val metrics, as expected (AbsRel 0.135, AUC@30 0.703 / 0.208 / 0.183).

VAll views (every pair and pixel of the V views)Shared views (only the first 2 views, the same in every V)ms per clipPeak GiB
AbsRel ↓δ1 ↑Rot. err.Rot. AUC@30Trans. AUC@30Pose AUC@30AbsRel ↓δ1 ↑Rot. err.Rot. AUC@30Trans. AUC@30Pose AUC@30
20.1110.86915.6°0.7000.2180.2100.1110.86915.6°0.7000.2180.21034.90.69
40.1220.87011.4°0.6830.2200.1910.1210.87311.6°0.7070.2540.23336.20.71
8 (trained)0.1350.85111.5°0.7030.2080.1830.1280.87011.7°0.7180.2380.21238.20.75
160.1670.80613.4°0.6720.1640.1480.1630.83113.5°0.7050.2150.19045.80.83

Pose pairs scored per scene: 1 / 6 / 28 / 120 for all views at V = 2 / 4 / 8 / 16, and always 1 for shared views. "Rot. err." is the mean rotation error. The model weights alone take 0.54 GiB of the peak memory.

Depth metrics, pose AUC at 30 degrees, forward time and peak memory against the number of input views
outputs/vggt_train/num_views/num_views.png. Solid lines score all views. Dashed lines ("first 2") score only the shared first two views. Forward time and peak memory are at batch size 1.

Conclusions

More views help pose, up to the training view count The shared-views columns score the same 2 views every time. So they show only the effect of extra context. Going from 2 to 4 views cuts the mean rotation error on that pair from 15.6° to 11.6°. It also raises translation AUC@30 from 0.218 to 0.254, and joint pose AUC@30 from 0.210 to 0.233. At 8 views the pose metrics are about the same as at 4. The extra frames give global attention more matching points (correspondences) to work out the relative pose from.
Depth gets worse with more views On the same two shared views, AbsRel rises steadily: 0.111, 0.121, 0.128 and 0.163 at V = 2, 4, 8 and 16. My best explanation is the normalization, because the metric has no scale alignment. The target scale is the centroid distance of the valid points over all V views. At V = 2, the model sees exactly the pixels that set the scale. At larger V, the depth of views 0 and 1 is measured in a unit set by all the views, which the model has to reason about together. So any error in that global scale shows up directly on every pixel. δ1 allows a 25% ratio error, and it stays at about 0.87 up to V = 8. That fits a mostly global scale error, not worse local shape.
At V = 16, every metric gets worse At 16 views, twice the training count, every metric is worse than at 8 in both scorings. On the shared views, AbsRel is 0.163 vs 0.128, and pose AUC@30 is 0.190 vs 0.212. Global attention now runs over twice as many tokens as it ever saw in training. So its softmax spreads over a longer sequence, and the token statistics change. The model does not automatically work for larger V. Training with a random V, as in Q2.3 for LVSM, would be the natural fix.
All views vs. shared views The all-views pose AUC falls faster with V than the shared-views AUC does (0.210 → 0.148 vs 0.210 → 0.190). This is because the scored pairs differ. With more views, more pairs come from frames far apart in the 32-frame window. Those pairs have larger relative motion, which is harder. So if you read only the all-views columns, you would overestimate how much extra views hurt pose.
Runtime and memory From 2 to 16 views, the forward pass grows from 34.9 to 45.8 ms (+31%). Peak memory grows from 0.69 to 0.83 GiB, and 0.54 GiB of that is weights. Global attention costs O((V·N)2) in compute. But at V = 16, the global sequence is only a little over 4,000 tokens. At batch size 1 and 224 px, the L40S is far from fully used. So the fixed overhead of each layer and the frozen DINOv2 encoder (linear in V) take most of the time, and the runtime grows much less than linearly. PyTorch's scaled_dot_product_attention also never builds the full attention matrix in memory. So memory above the weights stays small. It goes from 0.15 to 0.29 GiB for 8× as many views, because part of it is fixed workspace that does not depend on V.
Limitations
  • This is one checkpoint, one seed and the val split only.
  • The shared-views scores come from just 28 view pairs (one per scene). So AUC differences of a few hundredths are within noise. The clear trends are the rise in depth error and the drop at V = 16.
  • The view sets are nested and come from the same 32-frame window. So a larger V means denser coverage of the same part of the scene, not wider coverage.
  • Timing is latency at batch size 1 on one GPU, not throughput at a realistic batch size.