To resize the intrinsics I used the align_corners=False rule from the docstring, x_out + 0.5 = s (x_in + 0.5), which means x_out = s * x_in + 0.5 s - 0.5 on each axis. I wrote that as a 3 x 3 matrix [[sx, 0, 0.5 sx - 0.5], [0, sy, 0.5 sy - 0.5], [0, 0, 1]] and multiplied it on the left of K, so the focal lengths get scaled by s and the principal point gets scaled and shifted by half a pixel. For example a 300 x 300 crop with the principal point at 149.5 (the center) ends up at 111.5, which is exactly the center of the 224 x 224 image.
For unprojection I build the pixel grid [u, v, 1] with u as the column and v as the row, multiply it by K^-1, and scale it by the depth. K^-1 [u, v, 1] always has z = 1, so multiplying by the z-depth gives the camera space point directly (this is why it's z-depth and not distance along a unit ray). Then I move the points to world space with X_world = R X_camera + t from the camera-to-world pose.
For canonicalization I take camera 0's pose as the reference and multiply every pose by its inverse (in float64), so every camera is expressed in camera 0's frame and camera 0 becomes the identity. Then I unproject the depth with these new poses so the points are in camera 0's frame too, get the scene scale from compute_scene_scale (the distance from camera 0 to the centroid of all the valid points), and divide the camera translations, the depth and the points by it. I don't subtract the centroid, so camera 0 stays at the origin and the centroid ends up exactly 1 unit away.
Unprojection (left, original world frame in meters) and canonicalization (right, camera 0's frame in normalized units)

036bce3393, frame 4470. Before: first camera center (3.3463, 4.6765, 1.4680) m and valid-point centroid distance 6.668 m. After: first camera center (0, 0, 0) and centroid distance 1.000.

076c822ecc, frame 2890. Before: first camera center (1.5398, 2.7673, 1.5862) m and centroid distance 4.435 m. After: first camera center (0, 0, 0), scene scale 2.830 m, and centroid distance 1.000.

079a326597, frame 7430. Before: first camera center (3.7559, 2.0824, 1.5325) m and centroid distance 2.718 m. After: first camera center (0, 0, 0) and centroid distance 1.000.

07f5b601ee, frame 3630. Before: first camera center (3.0944, 3.4889, 1.4076) m and centroid distance 5.118 m. After: first camera center (0, 0, 0) and centroid distance 1.000.

07ff1c45bb, frame 460. Before: first camera center (1.3404, 2.1707, 1.5162) m and centroid distance 3.285 m. After: first camera center (0, 0, 0) and centroid distance 1.000.
In the unprojection plots the points from all 8 views line up into one room, which means the depth, the intrinsics and the poses all agree. The red frustum is camera 0, and before canonicalization it sits wherever the scan happened to put it (about 1.3 to 3.8 m from the world origin in these scenes). After canonicalization the same room is just moved and rotated so camera 0 is at the origin looking down +Z, and shrunk so the centroid of the points is 1 unit away, so a big room and a small room end up at about the same size. The few stray points far away from the room in some scenes are in the depth maps themselves (they show up in both stages), not from the unprojection.
For the rays I use the same pixel grid and K^-1 to get a direction in camera space for every pixel, rotate it into the canonical world frame with the rotation part of the camera-to-world pose (R d, with no translation since a direction doesn't move with the camera), and normalize it to unit length. The origin of every ray is just the camera center (the translation part of the pose), copied to every pixel. The Plücker coordinates are the 6 numbers [d, m] with the moment m = o x d.
The world space ray directions tell the model where each pixel is looking in the shared frame, so pixels from different views that look at the same point in the room have rays that cross there. The direction alone doesn't say where the ray is though, since parallel rays from different cameras have the same direction. The moment fixes that: m = o x d is perpendicular to the plane that contains the ray and the origin, and its length is the distance from the origin to the line. It also doesn't change if you slide the origin along the ray (o + t d gives the same m because d x d = 0), so [d, m] describes the line itself and not a specific point on it. Since the scene is canonicalized, the moments are in normalized units and have similar sizes across scenes.
python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
All four samples have the right shapes, the reprojection errors for the points and the rays are around 0.0001 pixels (way under 0.01), and the centroid distance is 1 up to float precision. The valid depth fraction is between 75% and 99%, and the 75% scene just has more missing depth (black pixels in the depth maps).
As an extra check I also ran my functions on the prepared validation and test clips, which were made by the course's own code. I took their K, poses and depth (scaled back to meters), ran them through canonicalize_cameras, camera_rays and plucker_coordinates, and compared to the stored values. Over 14 clips the biggest difference was 5e-7 for every output (scale, poses, depth, points, ray origins, ray directions and Plücker), which is just float32 rounding.
patchify reshapes the images to (L, C, gh, p, gw, p), permutes to (L, gh, gw, p, p, C) and flattens, so the patches go left to right then top to bottom and each patch is flattened by row, column, then channel. unpatchify does the exact same steps backwards, and I checked that unpatchify(patchify(x)) gives back x exactly. In build_tokens I add the same 2D sin-cos positional embedding to the source and target tokens, flatten the S source views into S * N tokens, repeat them for every target, and put that target's N query tokens at the end, so the transformer gets B * T sequences of (S + 1) * N tokens. In forward the source RGB is shifted to [-1, 1] and concatenated with the source Plücker rays (9 channels), patchified and projected with source_proj, and the target rays (6 channels) are patchified and projected with target_proj, so the target never sees any RGB. After input_norm and the 8 blocks I keep the last N tokens of each sequence, run them through out_norm, rgb_head and a sigmoid, and unpatchify back to (B, T, 3, H, W). The loss is the MSE over every pixel of every target view, and the training step runs the model under bfloat16 autocast but casts the prediction to float32 before the loss. I also checked that changing one target's rays doesn't change the prediction for another target, since each target gets its own sequence.
Final overfit PSNR = 30.92 dB (SSIM 0.870) after 2000 steps on one fixed batch

Each row is one target view and the columns are the 2 source views, the ground truth and the prediction. After 2000 steps on the same batch the predictions look almost exactly like the ground truth, even for the targets that look at a part of the room the source views barely show (rows 2 and 3), which is just the model memorizing this batch. This took about 4 minutes.
Best validation PSNR = 16.14 dB, SSIM = 0.560 (2 source views, epoch 99)
I used configs/lvsm.json as it is: patch size 8 (784 tokens per view), width 384, 8 layers, head dim 64 with QK norm (14.6M parameters), 8 views per sample with 2 source views and 6 targets, batch size 4, AdamW with learning rate 2e-4, weight decay 0.05 and betas (0.9, 0.95), 200 warmup steps then cosine decay to 10%, gradient clipping at 1, and bfloat16 autocast. It trained for 100 epochs of 56 steps (5,600 steps in total), which took 16 minutes on my RTX 5090.

The loss goes down and the PSNR goes up the whole time and both are still slowly improving at epoch 100, so training longer would help. The validation PSNR is actually a little higher than the training PSNR, so there isn't any overfitting. Part of that is because the training number is averaged over the epoch while the weights are still changing, and the validation clips are different scenes.
Validation during training at epochs 0, 50 and 95 (columns: 2 source views, ground truth, prediction)

At epoch 0 the predictions are just flat color. By epoch 95 they get the rough layout of the target view right, like where the bright window is and where the dark chair is, but they are very blurry and you can see the 8 x 8 patches. MSE loss rewards predicting the average when the model isn't sure, and with only 5,600 steps this small model is still underfitting (the training PSNR is about the same as validation).
Ablations (best.pt on the 28 validation scenes, 8 views from the first 32 frames)
| Source views | Conditioning | PSNR | SSIM | Target images |
|---|---|---|---|---|
| 1 | correct | 15.20 | 0.558 | 196 |
| 1 | shuffled | 11.81 | 0.488 | 196 |
| 1 | none | 12.47 | 0.528 | 196 |
| 2 | correct | 16.14 | 0.560 | 168 |
| 2 | shuffled | 11.49 | 0.472 | 168 |
| 2 | none | 12.45 | 0.527 | 168 |
| 4 | correct | 16.88 | 0.575 | 112 |
| 4 | shuffled | 11.59 | 0.473 | 112 |
| 4 | none | 12.47 | 0.537 | 112 |
Correct rays with 1, 2 and 4 source views (columns: source views, ground truth, prediction)

2 source views with shuffled rays (left) and no rays (right)

More source views help even though the model was only trained with 2. Going from 1 to 2 to 4 sources gives 15.20, 16.14 and 16.88 dB, since more of the room is visible and the target is more likely to overlap with one of the sources. It isn't a perfectly fair comparison because the targets are the views that aren't sources (7, 6 or 4 per clip), so the target sets are a bit different, but the trend is pretty big. SSIM barely changes from 1 to 2 sources though, so the extra source mostly fixes the colors and big shapes and not the fine structure, which is still blurry either way.
The target rays are what tell the model which view to render, and without them it falls apart. With no rays (all zeros) every target gets the same input, so the model can only output one image for all of them, and it learned to output a flat average color of the room (about 12.5 dB no matter how many sources there are). Shuffled rays are even worse (11.5 to 11.8 dB) because the model confidently renders the wrong view. In the shuffled image above each prediction looks like the correct prediction for the other target, so the model really is following the rays it is given and not just copying a source. The gap between correct and shuffled (over 4 dB) shows that almost everything the model gets right comes from using the target camera.
Question: does the number of source views used during training matter, and does a model trained with one count work with a different count at test time?
I trained three more models that are exactly the same as the baseline except for the number of source views in training: always 1, always 4, and a random number from 1 to 4 for every step (I added a train_source_views option to the config for this, and validation during training still uses 2). Every model trained for the same 100 epochs with the same seed and data order, and then I evaluated each best.pt with 1, 2 and 4 source views and the correct rays.
| Trained with | Eval 1 source | Eval 2 sources | Eval 4 sources | Seconds per epoch |
|---|---|---|---|---|
| 1 source | 15.12 / 0.559 | 14.91 / 0.553 | 14.64 / 0.561 | 7.7 |
| 2 sources (baseline) | 15.20 / 0.558 | 16.14 / 0.560 | 16.88 / 0.575 | 9.5 |
| 4 sources | 15.25 / 0.555 | 16.37 / 0.560 | 17.42 / 0.577 | 11.2 |
| random 1 to 4 | 15.26 / 0.559 | 16.22 / 0.559 | 17.09 / 0.575 | 9.6 |
The numbers are PSNR / SSIM on the validation scenes.

Evaluated with 4 source views: trained with 1 (left), 4 (middle) and random 1 to 4 (right)

The model trained with 1 source actually gets worse when you give it more views (15.12 to 14.64 dB). It never saw more than one source during training, so it never learned how to combine information from several views, and the extra source tokens probably just get in the way of its attention. Every model that saw at least 2 sources in training gets better with more sources at test time.
Training with 4 sources was the best with 2 and 4 sources at test time, and tied with the others with only 1 source (15.25 dB vs 15.20 to 15.26). My guess is that with more sources each training target has more of the room to copy from, so the model gets a clearer signal about how to match source patches to the target rays and learns that faster. It also helped most when the test setting matched training (17.42 dB with 4 sources). Training with a random count was in between: it beat the baseline at 2 and 4 sources and was about the same at 1, so it is a reasonable choice if you don't know how many views you'll have at test time.
There are some limitations. Each setting was trained once, so the differences of 0.1 to 0.2 dB at 1 source are probably just noise, although the bigger trends (1-source training getting worse with more views, and 4 sources being best at 2 and 4) are much larger than that. Training with more sources also costs more because the sequences are longer (11.2 s vs 7.7 s per epoch), and it means fewer targets per step (4 instead of 7), so it isn't only the number of sources that changes. All of these models are also still underfitting after 5,600 steps, so the ranking could change with much longer training.
In aggregate the blocks alternate between frame attention and global attention. For frame attention I reshape the tokens to (B * V, N, D) so each view only attends to itself and use the per-view 2D positions, and for global attention I reshape to (B, V * N, D) so every token can attend to every view, with the same positions repeated for each view (no view-order embedding, since camera 0 already has its own camera and register tokens). I keep the output of the last frame block, concatenate it with the output of the last global block to get 2D features, and apply the norm. The camera head runs its MLP on the camera token and outputs 9 numbers: the world-to-camera translation and quaternion stay linear and the two FoVs go through a ReLU, all in float32. The depth head projects every patch feature to a 14 x 14 depth patch, turns the patches back into an image, adds a residual from the small conv that looks at the depth and the RGB together, and takes exp so the depth is positive. The confidence is projected the same way and is 1 + exp(raw), so it's always at least 1. I compute both exps in float32 because the model runs in bfloat16.
The camera loss compares the predicted encoding directly to the GT encoding: L1 on the translation (clamped at 100), L1 on the quaternion and 0.5 times L1 on the FoVs. The depth loss is the trimmed mean of |pred - GT|, plus the trimmed mean of conf * |pred - GT| - 0.2 log(conf), plus a gradient loss. The confidence term lets the model lower the weight of pixels it's unsure about, but the log term stops it from just making every confidence tiny. The gradient loss takes the residual (pred - GT) at strides 1, 2, 4 and 8 and penalizes the difference between neighboring residuals where both pixels are valid, so the depth edges have to line up with the GT even if the overall scale is a bit off. I multiply by the masks instead of indexing so the loss stays connected to the graph even when nothing is valid. Before training I checked that the camera loss is exactly 0 at the GT encoding, the depth loss is 0 at the GT depth with confidence 1, the gradients are finite, and changing one view's image changes another view's depth (so the global attention really mixes views).

Ground truth (left) vs predicted camera + depth (right) on the overfit clip

I overfit one training clip (036bce3393, 8 views) with configs/vggt.json for 40 epochs of 25 steps at learning rate 1e-3, which took about 2.5 minutes. All three losses drop fast in the first few epochs and then keep going down slowly. The total loss goes below 0 at the end, which is fine because the confidence term has the -0.2 log(conf) part, so it can be negative once the depth error is small and the confidence is high. By the end the depth AbsRel went from 0.62 to 0.036, the depth delta1 from 0.21 to 0.956, the rotation error from 49.7 to 0.18 degrees and the translation error from 90 to 0.36 degrees (pose AUC at 5 degrees went from 0 to 0.98). In the viewer the predicted point cloud has the same shape as the GT and the predicted cameras sit in the same places, it's just a bit noisier around the edges where the depth jumps, so the model and the losses are working.
I trained with configs/vggt.json as it is: frozen DINOv2 ViT-B/14, width 384, 6 heads, 8 frame/global pairs, 8 views per sample from a 32-frame window, batch size 16, camera loss weight 5 and depth loss weight 1, AdamW with learning rate 6e-4 and weight decay 0.01, 10 warmup epochs then cosine decay to 5% of the learning rate, and 1000 epochs of 14 steps (14,000 steps in total), with bfloat16 and gradient clipping at 1. It took 2 hours 24 minutes on my RTX 5090, and best.pt is from epoch 879 (lowest validation loss, 0.794).

The depth loss drops fast in the first ~20 epochs and then improves slowly, but the camera loss keeps going down for the whole run and is still going down a bit at the end, so the cameras are the harder part to learn and more training would probably still help. The validation loss is lower than the training loss the whole time. The training loss is averaged over random 8-view windows from anywhere in the scans while validation always uses the first 32 frames of each clip, so they aren't exactly the same task, but there's no sign of overfitting.
Metrics (best.pt, 8 views from the first 32 prepared frames, 224px, seed 825, full splits)
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel (lower is better) | 0.135 | 0.157 |
| Depth delta1 | 0.832 | 0.793 |
| Rotation AUC @ 5 / 15 / 30 | 0.240 / 0.556 / 0.724 | 0.195 / 0.538 / 0.740 |
| Translation AUC @ 5 / 15 / 30 | 0.023 / 0.097 / 0.208 | 0.012 / 0.066 / 0.183 |
| Joint pose AUC @ 5 / 15 / 30 | 0.009 / 0.077 / 0.184 | 0.004 / 0.049 / 0.169 |
| Average rotation / translation direction error | 12.0 / 40.0 degrees | 8.9 / 40.4 degrees |
Each split has 28 scenes and 784 camera pairs, and none of the pairs had an undefined translation direction.
Ground truth (left) vs predicted camera + depth (right) on four validation scenes

09c1414f1b. The overall shape of the room and where the cameras are match the GT, but the walls come out smoother and a little wavy.

21d970d8de. The box shape of the room is there, but the thin things like the table legs and the sharp edges get smeared into a rounder cloud.

280b83fcf3. The main part of the room matches, but the sparse wall on the far right of the GT mostly isn't in the prediction.

2b1dc6d6a5. The cameras are roughly in the right places, but the long thin structures turn into blobs.
On held-out scenes the model is decent at depth (about 83% of validation pixels within 25% of the GT depth) and okay at rotation (an average error of 12 degrees, and over half of the pairs under 15 degrees), but it's bad at translation. The average translation direction error is about 40 degrees and the translation AUC at 15 degrees is under 0.1, and since the joint pose AUC uses the worse of the two errors, translation is what keeps it so low. I think translation is harder because the cameras in an 8-view window are close together, so the translation direction between two of them depends on a small amount of motion, while rotation can be read from how the whole image shifts. The point clouds show the same thing as the depth numbers: the coarse layout is right but fine structure is smoothed out, and since the predicted points come from predicted depth and predicted cameras, small camera errors also make the views not line up perfectly, which adds to the blur. The test split is a little worse than validation for depth (0.157 vs 0.135 AbsRel) and about the same for the cameras.
Question: how does the number of input views change the accuracy, runtime and memory of a model that was trained with 8 views?
I evaluated the same best.pt from 3.2 on the full validation split with 2, 4, 8, 16 and 32 views, always picked from the first 32 prepared frames with frame 0 as the reference and seed 825, so the only thing that changes is the number of views. I also timed the forward pass on one sample with bfloat16 (20 runs after 3 warmup runs, with nothing else on the GPU) and recorded the peak GPU memory. The code is in scripts/vggt_view_count.py.
| Views | Depth AbsRel | Depth delta1 | Rotation error | Translation error | Rotation AUC@15 | Translation AUC@15 | Pose AUC@30 | Forward time | Peak memory |
|---|---|---|---|---|---|---|---|---|---|
| 2 | 0.109 | 0.890 | 15.7 | 34.3 | 0.557 | 0.121 | 0.238 | 23.6 ms | 0.97 GB |
| 4 | 0.126 | 0.856 | 13.4 | 37.7 | 0.523 | 0.128 | 0.194 | 23.3 ms | 0.98 GB |
| 8 (trained) | 0.135 | 0.832 | 12.0 | 40.0 | 0.556 | 0.097 | 0.184 | 22.6 ms | 1.02 GB |
| 16 | 0.160 | 0.799 | 13.2 | 41.5 | 0.506 | 0.082 | 0.166 | 34.7 ms | 1.06 GB |
| 32 | 0.169 | 0.773 | 12.6 | 41.5 | 0.504 | 0.074 | 0.156 | 36.8 ms | 1.40 GB |
The errors are in degrees and averaged over all pairs (28, 168, 784, 3,360 and 13,888 pairs).

Giving the model more views than it was trained with makes everything worse. At 16 and 32 views the depth AbsRel goes up to 0.160 and 0.169 and all the pose AUCs drop. The global attention now runs over 2 or 4 times as many tokens as it ever saw in training, so each token's attention gets spread over many more views, and the features the heads get look different from what they were trained on. The rotation error is lowest right at 8 views, which is the one thing that peaks where the model was trained.
The surprising part is that fewer views is actually better for depth and translation: 2 views gets the best AbsRel (0.109) and the best pose AUC@30 (0.238). The frames are drawn from the same 32-frame window for every view count, so it isn't because the 2-view frames are easier or closer together. I think it's because of the scale normalization. The GT depth is divided by the distance to the centroid of all the selected views' points, so to get the depth right the model has to guess that shared scale from all the images, and that's easier when there are only two of them. So the extra views don't really help this model with depth, which is mostly decided from each image (the frozen DINO features and frame attention), and they make the shared scale harder to guess. I didn't test this directly though, so it's just my best explanation.
Runtime and memory barely change for one sample. From 2 to 8 views the forward pass takes about 23 ms no matter what, because with this few tokens the GPU isn't busy and the time is mostly fixed overhead. It only goes up at 16 and 32 views (35 to 37 ms). The memory goes from 0.97 to 1.40 GB, and most of that is the model weights (the ViT-B encoder alone is 86M parameters), since attention uses PyTorch's memory-efficient kernel and there are no gradients at evaluation. The attention cost grows with (views x 261 tokens)^2, so with bigger batches or training (which keeps activations for backprop) it would grow much faster.
The limitations are that this is one model trained with exactly 8 views, so it can't tell whether a model trained with more views would actually use them better, and that metrics over different numbers of views aren't perfectly comparable (2 views only gives 28 pairs, so its pose AUCs are noisier). A model trained with a random number of views each step, like I tried for LVSM in 2.3, would be the natural next experiment.