Contents
- Multi-View Data Preprocessing: visualizations · final check · canonicalization, rays and Plücker coordinates
- Novel-View Synthesis with Mini-LVSM: 2.1 · 2.2 · 2.3 (bonus)
- Multi-View Geometry with Mini-VGGT: 3.1 · 3.2 · 3.3 (bonus)
1. Multi-View Data Preprocessing
All Part 1 code is in a4data/geometry.py. I follow the conventions of the data
contract. Cameras use OpenCV axes (+X right, +Y down, +Z forward). Poses are camera-to-world:
Xworld = R Xcam + t.
1.2–1.3 Data Visualizations
Both pages come from scripts/visualize_data.py --num-scenes 5 --max-points 10000.
They show the same five training clips (8 views each, seed 825). The left images are after
unprojection. They are in the original ScanNet++ world frame, in meters. The right images are
after canonicalization. They are in camera 0's frame (the red frustum). The side panel shows
view 0's RGB and depth, the first camera center, and the distance to the centroid of the valid
points. Click an image to see it at full size.
| Clip | After unprojection (m) | After canonicalization | |||
|---|---|---|---|---|---|
| First camera center | Centroid distance | First camera center | Scene scale s | Centroid distance | |
036bce3393frame 4470 | (3.3463, 4.6765, 1.4680) | 6.668 | (0, 0, 0) | 1.867 m | 1.000 |
076c822eccframe 2890 | (1.5398, 2.7673, 1.5862) | 4.435 | (0, 0, 0) | 2.830 m | 1.000 |
079a326597frame 7430 | (3.7559, 2.0824, 1.5325) | 2.718 | (0, 0, 0) | 2.080 m | 1.000 |
07f5b601eeframe 3630 | (3.0944, 3.4889, 1.4076) | 5.118 | (0, 0, 0) | 0.924 m | 1.000 |
07ff1c45bbframe 460 | (1.3404, 2.1707, 1.5162) | 3.285 | (0, 0, 0) | 1.966 m | 1.000 |
The numbers are as the viewer prints them. Before canonicalization, the centroid distance is only the distance from ScanNet++'s arbitrary world origin. After canonicalization, camera 0 is at the origin (the viewer prints ±0.0000). The centroid is 1.000 units away in every clip. The scale that was divided out ranges from 0.92 to 2.83 m. In both stages, the eight views fit together into one consistent room.

036bce3393, after unprojection
036bce3393, after canonicalization
076c822ecc, after unprojection
076c822ecc, after canonicalization
079a326597, after unprojection
079a326597, after canonicalization
07f5b601ee, after unprojection
07f5b601ee, after canonicalization
07ff1c45bb, after unprojection
07ff1c45bb, after canonicalization1.7 Final Check
python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
on the full training split (224 clips):
| Sample | Views | Resolution | Valid depth | Max point reprojection error | Max ray reprojection error | Centroid distance |
|---|---|---|---|---|---|---|
e050c15a8dframe 7080 | 8 | 224 | 97.8% | 1.13e-4 px | 1.09e-4 px | 1.0 |
5f99900f09frame 2820 | 8 | 224 | 99.0% | 1.23e-4 px | 1.11e-4 px | 0.99999994 |
47b37eb6f9frame 2720 | 8 | 224 | 96.1% | 9.88e-5 px | 8.90e-5 px | 1.0 |
4c5c60fa76frame 6690 | 8 | 224 | 75.2% | 1.09e-4 px | 1.03e-4 px | 0.99999994 |
Batch shapes (batch size 2 from data.json, V = 8 views, 224×224):
| Key | Shape | Key | Shape |
|---|---|---|---|
rgb | (2, 8, 3, 224, 224) | points_world | (2, 8, 3, 224, 224) |
depth | (2, 8, 1, 224, 224) | ray_origins | (2, 8, 3, 224, 224) |
depth_valid | (2, 8, 1, 224, 224) | ray_directions | (2, 8, 3, 224, 224) |
K | (2, 8, 3, 3) | plucker | (2, 8, 6, 224, 224) |
T_world_from_camera | (2, 8, 4, 4) | frame_numbers | (2, 8) |
original_world_from_reference | (2, 4, 4) | scene_scale | (2,) |
resize_intrinsics. Pixel centres are at integers, so a resize by s maps
c → s·c + (s − 1)/2, not s·c. For the 300 px crop resized to 224,
leaving out that offset would shift every pixel by about 0.13 px.
1.3–1.5 Canonicalization, Ray Directions and the Plücker Moment
Why: each ScanNet++ scene has its own arbitrary world frame, and rooms differ in size. From images alone, you cannot work out a global rigid transform or a global scale. Canonicalization removes both, so a network never has to guess them. The evaluation loader does the same thing.
World-space ray directions Pixel p = (u, v, 1) looks along K−1p in the camera frame. To get the world direction, I rotate that by the camera-to-world rotation and normalize it:
The Plücker moment Writing a ray as (o, d) is redundant, because any point on the line could be o. Plücker coordinates [d, m] with m = o × d remove that extra freedom. If I slide the origin to o + λd, I get (o + λd) × d = o × d, because d × d = 0. So m depends only on the line. It is perpendicular to d. When d is a unit vector, the length of m is the distance from the origin to the line. d says which way the ray points, and m says where the line is. So a network gets each pixel's ray as 6 channels with no ambiguity. Camera 0's rays pass through the origin, so their moments are 0. Rescaling by s scales the moments by 1/s, the same as the origins.
2. Novel-View Synthesis with Mini-LVSM
All Part 2 code is in models/lvsm/. Mini-LVSM (14.6M parameters) predicts a
target image from S source images and the target camera's Plücker rays. It never sees the
target's RGB. Each source patch has RGB (shifted to [−1, 1]) plus rays, so 9 channels.
Each target query patch has only the 6 ray channels. patchify cuts 8×8
patches (a 28×28 grid, so N = 784 per view). Two linear layers embed them. The same 2D
sin-cos position embedding is added to every view. Each target gets its own sequence: all
S·N source tokens, then its own N query tokens. So targets never see each other. After 8
full-attention blocks, I keep only the query tokens. I decode them with the RGB head and a
sigmoid, then run unpatchify. The loss is MSE. I compute it in float32, outside the
bf16 autocast.
2.1 Overfitting
python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit trains
for 2000 steps on one fixed training batch (4 clips × 6 targets = 24 images). Then it
scores that same batch.
| Final PSNR | Final SSIM | Final MSE | Mean train PSNR over the 2000 steps |
|---|---|---|---|
| 30.90 dB | 0.870 | 0.00087 | 25.37 dB |
outputs/lvsm_overfit/val_epoch000.png: source view 1 | source view 2 |
ground truth | predictionsave_visualization takes the first clip in the batch. It draws one row per target
view (4 of its 6). Each row shows the same two source views, then the ground-truth target, then
my prediction. The predictions match the ground truth closely. This is true even for the two
rows that face the dark-blue wall, which neither source shows. This is memorization, not
novel-view synthesis, because here the "validation" batch is the training batch. Faint
8×8 patch edges are still visible, because each query token decodes its own patch. The
mean train PSNR is lower because it includes the early steps.
2.2 Training and Evaluation
Training settings
| Setting | Value |
|---|---|
| Configs | configs/lvsm.json with configs/scannetpp/iphone/data.json |
| Training data | 224 training scenes; 8 views per sample from a 32-frame window; 224×224 |
| Views | S = 2 sources and T = 6 targets per sample |
| Batch and length | B = 4, so 56 steps per epoch; 100 epochs = 5,600 steps |
| Model | patch 8, dmodel 384, 8 layers, head dim 64, MLP ratio 4, QK-norm |
| Optimizer | AdamW, lr 2e-4, weight decay 0.05, betas (0.9, 0.95), gradient clip 1.0 |
| Schedule | 200 warmup steps, then cosine decay to 10% of the peak |
| Precision | bf16 autocast (AMP), loss in float32 |
| Validation | 28 val scenes, 8 fixed views from the first 32 prepared frames, S = 2 (168 target images); best.pt = best val PSNR |
| Hardware | one NVIDIA L40S, about 29 minutes |
Training curves
outputs/lvsm/log.jsonl. Train values are averaged over the
steps of each epoch. Validation uses the full val split at S = 2.
val_epoch095.png, the last one saved (one is
saved every 5 epochs): source 1 | source 2 | ground truth | predictionAblations: number of source views and ray conditioning
python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none
on the val split. best.pt is from the last epoch (99). Shuffled gives
each target the rays of a different target (shifted by one). None sets all target rays
to zero. Every S uses the same 8 views per scene. Only the split into sources and targets
changes.
| S | Targets (images) | Correct | Shuffled | None | |||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | ||
| 1 | 7 per scene (196) | 15.19 dB | 0.558 | 11.80 dB | 0.487 | 12.46 dB | 0.529 |
| 2 (trained) | 6 per scene (168) | 16.14 dB | 0.560 | 11.49 dB | 0.472 | 12.44 dB | 0.528 |
| 4 | 4 per scene (112) | 16.86 dB | 0.575 | 11.59 dB | 0.473 | 12.46 dB | 0.538 |
These numbers are within about 0.2 dB of the course staff's reference solution with the same config (Piazza @89: about 15, 16 and 17 dB PSNR and 0.56, 0.56 and 0.57 SSIM at S = 1, 2, 4).
Evaluation images for the first val clip (source views | ground truth | prediction):
Discussion
Ray conditioning The target rays are the only thing that tells the model where the target camera is. They matter a lot. At S = 2, correct rays give 16.14 dB, no rays give 12.44 dB and shuffled rays give 11.49 dB. Without the right rays, more sources do not help at all (the none and shuffled columns stay flat across S). In the images, no rays gives an almost flat beige image. Every query token is the same except for its position, so the model falls back to the scene's average colour. Shuffled rays give a blurry rendering from another target's camera.
Why shuffled rays are worse than none MSE rewards safe, averaged guesses. With no pose, the best the model can do is the average colour. That is the best constant guess under MSE. With a wrong pose, the model commits to a layout (dark chairs, a bright window) but puts it in the wrong places. So it is penalized for the structure it adds and also for the structure it misses. If the wrong view had nothing in common with the target, the error would be about double the error of the flat guess (3 dB). The actual gap is 0.7–1.0 dB. I think this is because the shuffled renders are blurry and neighbouring views overlap. SSIM shows the same thing (0.472 vs 0.528 at S = 2), because its structure term penalizes wrong edges.
Blurry results and undertraining The predictions are blurry and blocky, at about 16 dB at S = 2. They get the rough layout and colours (window, table, dark chair) but no detail. The 8×8 patch grid is visible. I think the main reason is the short default schedule. 5,600 steps was not enough. Validation PSNR was still rising at the end (15.95, 16.03 and 16.06 dB averaged over epochs 70–79, 80–89 and 90–99). The best checkpoint is also the last epoch. Train and validation stay close the whole time (15.60 vs 16.14 dB at the end), so the model is underfitting, not overfitting. That is why I tried a 4× longer schedule, below.
Training longer: 400 epochs
The Piazza answer above confirms that the model is underfit. It also allows longer training
if the settings are stated. configs/lvsm_long.json is configs/lvsm.json
with 400 epochs instead of 100 (22,400 steps), and it saves validation images every 20 epochs.
The cosine schedule now spans all 400 epochs. Data, model, optimizer, peak learning rate, warmup
and seed are unchanged. It took 97 minutes on one L40S. That run had 8 CPUs and the 100-epoch
run had 4, so the wall times are not directly comparable. best.pt is epoch 378. I
evaluated it with the same command as above plus --lpips (AlexNet LPIPS, lower is
better). For the 100-epoch LPIPS values, I reran that model's evaluation with
--lpips.
| S | 100 epochs (above) | 400 epochs | Δ PSNR | ||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS ↓ | PSNR | SSIM | LPIPS ↓ | ||
| 1 | 15.19 dB | 0.558 | 0.705 | 15.91 dB | 0.588 | 0.659 | +0.72 dB |
| 2 (trained) | 16.14 dB | 0.560 | 0.673 | 17.20 dB | 0.599 | 0.607 | +1.07 dB |
| 4 | 16.86 dB | 0.575 | 0.652 | 18.39 dB | 0.621 | 0.583 | +1.53 dB |
Correct rays. With shuffled target rays the
400-epoch model gets 11.2–11.6 dB, and with no target rays it gets 12.4–12.5 dB.
This is the same as the 100-epoch model
(outputs/lvsm_long/eval_val/metrics.json).
log.jsonl. Validation is at S = 2.Where the gain comes from With shuffled rays or no rays, the longer model is no better than the short one. So the extra quality comes from making better use of the target pose, not from a better average guess.
No more gain at this model size Validation PSNR averaged 17.17, 17.16 and 17.17 dB over the last three 10-epoch windows, while the learning rate decayed to 10% of its peak. So the model has stopped improving under this schedule. Train PSNR (16.60 dB at the end, on random clips) is still below validation, so the model is still not overfitting. The next step would be more capacity, such as a wider or deeper model, or smaller patches. The Piazza answer lists these as bonus directions.
2.3 Bonus: Training View Count, and a Perceptual Loss
Settings
configs/lvsm_random_views.json is configs/lvsm.json plus
"num_source_views_range": [1, 4]. Each training step draws S uniformly from 1 to 4
for the whole batch. The other 8 − S views are targets. Everything else is the same:
data, model, optimizer, schedule, 100 epochs and seed. Validation during training stays at
S = 2, so the two curves can be compared directly. After training, I evaluated both
best.pt files (epoch 99 in both runs) with --num-source-views 1 2 4
and correct rays.
Results
| Eval S | Fixed S = 2 (baseline) | Random S ∈ [1, 4] | Δ PSNR | ||
|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | ||
| 1 | 15.19 dB | 0.558 | 15.27 dB | 0.559 | +0.08 dB |
| 2 | 16.14 dB | 0.560 | 16.22 dB | 0.560 | +0.09 dB |
| 4 | 16.86 dB | 0.575 | 17.10 dB | 0.575 | +0.25 dB |
Δ is computed before rounding. Wall time was 28.9 vs 29.2 min, and peak GPU memory was 11,050 vs 12,256 MiB (baseline vs random S, from the job log).
log.jsonl, S = 2).
Right: evaluation PSNR vs S (eval_val/metrics.json, correct rays).



The two models are hard to tell apart by eye. Both are as blurry as in 2.2.
Conclusions
Why it might help The baseline never saw more than 2 source views. So at S = 4, its attention gets a longer sequence than it trained on. For the random-S model, S = 1 and S = 4 are part of training. It only spent about a quarter of its steps at S = 2, and it still matched the baseline there. My guess is that mixing S works a bit like data augmentation, because each scene is seen with different amounts of context. I did not test that.
Limitations
- One seed per setting, and small gains (+0.08 / +0.09 / +0.25 dB). So I cannot rule out run-to-run noise. A steady lead over 20 epochs shows that the two runs differ. It does not show why.
- Train PSNR is not comparable. The random-S run's training PSNR mixes S = 1–4. So I only compare validation curves and evaluations.
- Both models are undertrained (see 2.2). The ranking could change with a longer schedule.
- Cost. Steps with S = 3 or 4 have longer sequences, so peak memory went up by about 1.2 GB. The wall time barely changed (29.2 vs 28.9 min). This is because the data loader is the bottleneck. In both runs the GPU was only about 60% busy, while the instance's 4 CPUs were 82% busy.
Second exploration: a perceptual loss
configs/lvsm_lpips.json is configs/lvsm.json plus
"loss": {"lpips_weight": 0.05, "lpips_net": "vgg"}. So the training loss is
MSE + 0.05 · LPIPSVGG over every target image (MSEPlusLPIPS in
models/lvsm/losses.py). The LPIPS network is pretrained and frozen. It runs in
float32, outside the bf16 autocast. Its input images are mapped from [0, 1] to [−1, 1].
The weight 0.05 is the one that pixelSplat and MVSplat use. At the start of training, the
weighted LPIPS term was 0.035 and the MSE was 0.065, so the two terms start at a similar size.
Data, model, schedule, 100 epochs and seed are the same as the baseline. The validation loss is
still plain MSE. best.pt is still picked by validation PSNR (epoch 99, the last
epoch). For evaluation I report LPIPS with the AlexNet backbone, which is the usual metric. This
way the model is not scored by the same network it trained against. Wall time was 35 min on the
8-CPU instance. Peak GPU memory was 15,136 MiB, against 11,050 MiB for the baseline.
| Eval S | MSE (baseline) | MSE + 0.05 LPIPS | Δ PSNR | ||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS ↓ | PSNR | SSIM | LPIPS ↓ | ||
| 1 | 15.19 dB | 0.558 | 0.705 | 14.99 dB | 0.515 | 0.711 | −0.20 dB |
| 2 (trained) | 16.14 dB | 0.560 | 0.673 | 15.79 dB | 0.512 | 0.689 | −0.34 dB |
| 4 | 16.86 dB | 0.575 | 0.652 | 16.29 dB | 0.519 | 0.682 | −0.56 dB |
Correct rays, full val split. The LPIPS run's
best.pt and last.pt give identical numbers because both are epoch 99.


Conclusions (perceptual loss)
Why I think it happens At about 16 dB, the model cannot predict where the real edges are. My guess is that it can still lower the VGG distance without knowing that. It does this by adding generic high-frequency texture inside each patch. That texture is in the wrong place, so MSE and SSIM get worse. AlexNet is a different network, and it does not reward the texture either. The curves fit this. The weighted LPIPS term only fell from 0.035 to 0.028 over training. Even so, validation PSNR and SSIM stayed below the baseline for the whole run. The run's final train MSE was also higher (0.0357 vs 0.0342).
Limitations
- One weight and one seed. A smaller weight might behave differently. So might training with MSE first and adding LPIPS later. I did not test either.
- A short schedule. Both runs are 100 epochs long. At that length the model is still undertrained (see the 400-epoch run in 2.2). A perceptual loss may only help once the model can put edges in the right place.
- The patch decoder may matter. Each 8×8 patch is decoded on its own. That could make it cheap to add the same texture everywhere. I did not test this.
3. Multi-View Geometry with Mini-VGGT
All Part 3 code is in models/vggt/. Mini-VGGT takes V RGB images and predicts
every camera and a depth map for each view, in one forward pass. The point clouds come from
unprojecting the predicted depth with the predicted cameras.
Alternating attention (
aggregate)
The 16 blocks alternate between two types. Frame blocks attend within one image, with shape
(B·V, N, D). They build context within each image. Global blocks attend across all
views, with shape (B, V·N, D). This is where cross-view matching, relative pose and a
shared scale come from. The heads take LayerNorm(concat(last frame output, last global
output)). So I save the frame output before the last global block overwrites it.
Heads The camera head runs 4 more blocks over the camera tokens. Then an MLP outputs translation (3), a quaternion (4) and the field of view (2, through a ReLU). Its last layer is initialized to give the identity rotation, zero translation and a 53° FoV. The depth head maps each patch token to 14×14 values. Then a small conv net on [coarse depth, RGB] adds a residual that starts at zero. This residual lines up depth edges with image edges. Depth = exp(·) and confidence c = 1 + exp(·) ≥ 1.
Losses The camera loss (weight 5) is L1 on translation, quaternion and 0.5 × FoV against the GT pose encoding. The depth loss (weight 1) has three parts: a trimmed L1 term, a confidence-weighted term c|e| − 0.2 log c, and a gradient loss on the residual at strides 1, 2, 4 and 8. The gradient loss rewards correct local shape. The confidence term is a Laplace likelihood. The network learns to give high confidence where it expects small errors.
3.1 Overfitting
python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit
fits one fixed training clip for 40 epochs × 25 steps. The clip is
036bce3393_frame_004470, the first clip from Part 1, with 8 views. Training uses
lr 1e-3, no warmup and cosine decay to 5%.
outputs/vggt_overfit/loss_curve.png: total, camera and depth loss on
the clip after each epoch (epoch 0 is before training)| Metric (on the clip) | Before training | After 40 epochs |
|---|---|---|
| Total loss | 5.48 | −0.169 |
| Camera loss (unweighted) | 0.342 | 0.0025 |
| Depth loss (unweighted) | 3.77 | −0.181 |
| Mean rotation error | 49.7° | 0.22° |
| Mean translation-direction error | 90.0° | 0.42° |
| Depth AbsRel | 0.619 | 0.036 |
| Depth δ1 | 0.208 | 0.956 |
| Rotation AUC @5 / 15 / 30 | 0 / 0.050 / 0.142 | 1.000 / 1.000 / 1.000 |
| Translation AUC @5 / 15 / 30 | 0 / 0 / 0 | 0.986 / 0.995 / 0.998 |
| Joint pose AUC @5 / 15 / 30 | 0 / 0 / 0 | 0.986 / 0.995 / 0.998 |
From the overfit_train entries of history.jsonl (28 view pairs).
The translation error starts at exactly 90°. This is because the camera head starts at zero
translation, and the metric gives a zero baseline an error of 90°.
outputs/vggt_overfit/visualization/index.html (default
camera, view 0). The predicted cloud has the same shape, scale and camera layout as the GT.
The model learned to put camera 0 at the origin. Its predicted center is (0.0002,
−0.0000, 0.0002), and the centroid distance is 0.999. The predicted depth map is
smoother than the GT and shows patch-sized blocks. The cloud also has more stray points along
depth edges.
3.2 Training and Evaluation
Training settings
| Setting | Value |
|---|---|
| Config | configs/vggt.json (defaults) |
| Training data | 224 training scenes; batch 16 clips × 8 views from a 32-frame window (sampling_range 32); 224×224; 14 steps per epoch |
| Model | frozen DINOv2 ViT-B/14; 8 frame/global block pairs, width 384, 6 heads; camera head with 4 extra blocks; depth head with RGB refinement |
| Optimizer | AdamW, lr 6e-4, weight decay 0.01, gradient clip 1.0 |
| Schedule | 1000 epochs; 10 linear warmup epochs, then cosine decay to 5% of the peak |
| Loss weights | camera 5, depth 1 |
| Precision and loading | bf16; 4 data workers (4 was faster than 8 in a short benchmark on this 4-CPU instance) |
| Hardware | one NVIDIA L40S; 5.6 h of training wall time, in two
segments. Epochs 0–33 took 16 min with outputs on the EBS root disk. The run then
resumed from last.pt for epochs 34–1000, which took 320 min (19.9 s per
epoch), with outputs moved to the instance's local NVMe disk. Each checkpoint is about 1 GB,
and rewriting last.pt every epoch at the root disk's 125 MB/s was adding 8–16 s
per epoch. The resume continues the same 1000-epoch schedule; the final learning rate is
3.0e-5, 5% of the peak as planned. The TA's reference took about 4.4 h on one A100. |
| Evaluation | best.pt (lowest val loss); 8 views from the first 32 prepared frames, 224 px, seed 825; full val and test splits (28 scenes each) |
Loss curve
outputs/vggt_train/loss_curve.png: total, camera and depth loss per
epoch, train and validation (epoch 0 is before training)| Val, after epoch | Total loss | Camera loss | Depth loss | AbsRel | Mean rotation error | Mean translation error | Pose AUC@30 |
|---|---|---|---|---|---|---|---|
| 0 (init) | 4.08 | 0.206 | 3.05 | 0.726 | 34.5° | 90.0° | 0.000 |
| 100 | 1.20 | 0.172 | 0.340 | 0.171 | 26.9° | 56.0° | 0.080 |
| 500 | 0.935 | 0.127 | 0.299 | 0.161 | 15.3° | 45.0° | 0.118 |
888 (best.pt) | 0.794 | 0.110 | 0.247 | 0.135 | 11.5° | 40.7° | 0.183 |
| 1000 (last) | 0.816 | 0.112 | 0.257 | 0.141 | 11.6° | 41.0° | 0.175 |
Validation and test metrics
These are the metrics of best.pt (epoch 888), from
outputs/vggt_train/metrics.json (val) and test_metrics.json (test).
Each split has 28 scenes with 8 views each. That gives 28 × 28 = 784 ordered view pairs
for the pose AUCs. There is no scale or shift alignment. So depth is compared directly in the
normalized units of Part 1.
| Split | AbsRel ↓ | δ1 ↑ | Rotation AUC ↑ | Translation AUC ↑ | Joint pose AUC ↑ | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| @5 | @15 | @30 | @5 | @15 | @30 | @5 | @15 | @30 | |||
| Validation | 0.135 | 0.851 | 0.176 | 0.507 | 0.703 | 0.017 | 0.092 | 0.208 | 0.002 | 0.063 | 0.183 |
| Test | 0.161 | 0.779 | 0.181 | 0.525 | 0.729 | 0.017 | 0.085 | 0.183 | 0.003 | 0.058 | 0.164 |
Mean errors, val / test: rotation 11.5° / 9.4°, translation direction 40.7° / 42.4°, normalized depth RMSE 0.177 / 0.207.
GT vs. predicted geometry on four validation scenes
Screenshots of outputs/vggt_train/visualization/index.html, which the val
evaluation of best.pt writes (default camera, view 0). Each row is one validation
scene. The ground truth is on the left. The right shows the predicted cameras, with the
predicted depth unprojected using those cameras.

09c1414f1b (dining room): ground truth
09c1414f1b: predicted camera + depth
21d970d8de (classroom): ground truth
21d970d8de: predicted camera + depth
280b83fcf3 (hallway): ground truth
280b83fcf3: predicted camera + depth
2b1dc6d6a5 (office): ground truth
2b1dc6d6a5: predicted camera + depth- Dining room. The table, chairs and wall are in the right place at the right scale (predicted centroid distance 0.973, GT 1.000). But the chairs and table blur together, and the thin chair legs are missing from the predicted depth map. The predicted cameras are bunched together. In the GT, view 1 is clearly off to the side.
- Classroom. The shape of the room, the floor and the row of desks are right. But the sharp desk edges and chair rows in the GT turn into one smooth, curved surface.
- Hallway. The corridor's length and direction are right. So is the shoe rack near the camera. The far end of the corridor is smeared out into a cone of points.
- Office. This scene has the clearest camera motion, a sweep to the right. The predicted frusta follow it in the right order, with roughly the right spacing. The office chair loses its outline in the depth map. The desk cloud is thicker and noisier than in the GT.
Errors on held-out scenes
- The 8 views come from a 32-frame window of a handheld iPhone walk-through. So the baselines (the distances between cameras) are often small. With a small baseline, a few centimetres of error in the camera center is a large error in direction. The bunched cameras in the dining-room scene are an example.
- The camera loss was still falling at epoch 1000. But 300 more epochs (below) barely changed the pose AUCs. So I don't think training time is the main limit.
best.pt by
val loss favours val slightly. But each split has only 28 scenes. The error also depends a lot
on the scene (room size, and how much of the view is wall or floor with no texture). So
differences between scenes can explain these gaps.
Training longer: 300 more epochs
To test whether more training helps, I fine-tuned best.pt for 300 more epochs
with configs/vggt_ft8.json. Data, model and loss weights are the same. The weights
are loaded with --init-checkpoint, which loads only the weights and starts a fresh
optimizer. Then there is a new 5-epoch warmup to lr 2e-4 (a third of the original peak) and
cosine decay to 5%. It took 85 minutes (16.3 s per epoch) on one L40S, with 6 data workers on an
8-CPU instance. The fine-tune's best.pt is epoch 176. Everything else on this page
still uses the original epoch-888 checkpoint.
outputs/vggt_ft8/loss_curve.png. Epoch 0 is the original
best.pt before fine-tuning.| Split and model | AbsRel ↓ | δ1 ↑ | Rotation AUC ↑ | Translation AUC ↑ | Joint pose AUC ↑ | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| @5 | @15 | @30 | @5 | @15 | @30 | @5 | @15 | @30 | |||
| Val, original | 0.135 | 0.851 | 0.176 | 0.507 | 0.703 | 0.017 | 0.092 | 0.208 | 0.002 | 0.063 | 0.183 |
| +300 epochs | 0.137 | 0.858 | 0.203 | 0.518 | 0.705 | 0.017 | 0.090 | 0.211 | 0.002 | 0.060 | 0.182 |
| Test, original | 0.161 | 0.779 | 0.181 | 0.525 | 0.729 | 0.017 | 0.085 | 0.183 | 0.003 | 0.058 | 0.164 |
| +300 epochs | 0.158 | 0.787 | 0.179 | 0.514 | 0.717 | 0.011 | 0.068 | 0.181 | 0.003 | 0.048 | 0.165 |
3.3 Bonus: Number of Input Views
Settings
python scripts/vggt_num_views.py --checkpoint outputs/vggt_train/best.pt --split val --eval-root data/scannetpp/prepared_eval --eval-num-frames 32 --image-size 224 --seed 825 --views 2 4 8 16 --batch-size 1 --device cuda --output outputs/vggt_train/num_views
- One checkpoint. I evaluate the Q3.2
best.pton the val split (first 32 prepared frames, 224 px, seed 825). Nothing is retrained. - Nested view sets. The first view is frame 0. The rest follow a fixed order (a permutation) of the 32 frames, chosen per scene. So each smaller set is the start of every larger set.
- Two ways of scoring. "All views" scores every pair and pixel of the V views. So the pairs change with V (from 1 pair per scene at V = 2, through 6 and 28, to 120 at V = 16). "Shared views" scores only the first 2 views, which every V includes. So the scored pairs and pixels stay fixed, and only the extra context changes.
- Scale. The scene scale is recomputed for each V, because the centroid of the valid points depends on which views are included. AbsRel, δ1 and the AUCs are ratios or angles, so they can still be compared across V.
- Runtime and memory use batch size 1 in bf16. Runtime is one synchronized forward pass, timed after one warm-up pass. Memory is the peak allocated CUDA memory.
Results
From outputs/vggt_train/num_views/num_views.json (val, 28 scenes). The V = 8
"all views" row matches the Q3.2 val metrics, as expected (AbsRel 0.135, AUC@30 0.703 / 0.208 /
0.183).
| V | All views (every pair and pixel of the V views) | Shared views (only the first 2 views, the same in every V) | ms per clip | Peak GiB | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AbsRel ↓ | δ1 ↑ | Rot. err. | Rot. AUC@30 | Trans. AUC@30 | Pose AUC@30 | AbsRel ↓ | δ1 ↑ | Rot. err. | Rot. AUC@30 | Trans. AUC@30 | Pose AUC@30 | |||
| 2 | 0.111 | 0.869 | 15.6° | 0.700 | 0.218 | 0.210 | 0.111 | 0.869 | 15.6° | 0.700 | 0.218 | 0.210 | 34.9 | 0.69 |
| 4 | 0.122 | 0.870 | 11.4° | 0.683 | 0.220 | 0.191 | 0.121 | 0.873 | 11.6° | 0.707 | 0.254 | 0.233 | 36.2 | 0.71 |
| 8 (trained) | 0.135 | 0.851 | 11.5° | 0.703 | 0.208 | 0.183 | 0.128 | 0.870 | 11.7° | 0.718 | 0.238 | 0.212 | 38.2 | 0.75 |
| 16 | 0.167 | 0.806 | 13.4° | 0.672 | 0.164 | 0.148 | 0.163 | 0.831 | 13.5° | 0.705 | 0.215 | 0.190 | 45.8 | 0.83 |
Pose pairs scored per scene: 1 / 6 / 28 / 120 for all views at V = 2 / 4 / 8 / 16, and always 1 for shared views. "Rot. err." is the mean rotation error. The model weights alone take 0.54 GiB of the peak memory.
outputs/vggt_train/num_views/num_views.png. Solid lines score all
views. Dashed lines ("first 2") score only the shared first two views. Forward time and peak
memory are at batch size 1.Conclusions
scaled_dot_product_attention also never builds the full attention matrix in
memory. So memory above the weights stays small. It goes from 0.15 to 0.29 GiB for 8×
as many views, because part of it is fixed workspace that does not depend on V.
- This is one checkpoint, one seed and the val split only.
- The shared-views scores come from just 28 view pairs (one per scene). So AUC differences of a few hundredths are within noise. The clear trends are the rise in depth error and the drop at V = 16.
- The view sets are nested and come from the same 32-frame window. So a larger V means denser coverage of the same part of the scene, not wider coverage.
- Timing is latency at batch size 1 on one GPU, not throughput at a realistic batch size.











