16-825 · Learning for 3D Vision
Intrinsics are resized with the pixel center convention: fx and fy are scaled by the width and height ratios, and the principal point becomes s (c + 0.5) - 0.5. Depth is unprojected as z K-1 [u, v, 1]T with u, v the integer pixel indices, and the camera points are moved to the world with the rotation and translation of T_world_from_camera.






In the world coordinates the first camera sits wherever the ScanNet++ scan put it, a few meters from the origin. After canonicalization the red frustum of view 0 is at the origin looking down +Z, and the same scene is shown in normalized units.
Train split, 224 samples, 8 views at 224 x 224:
| Sample | Valid depth | Max point reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.978 | 1.13e-4 | 1.09e-4 | 1.000 |
| 5f99900f09_frame_002820 | 0.990 | 1.23e-4 | 1.11e-4 | 1.000 |
| 47b37eb6f9_frame_002720 | 0.961 | 9.88e-5 | 8.90e-5 | 1.000 |
| 4c5c60fa76_frame_006690 | 0.752 | 1.16e-4 | 1.07e-4 | 1.000 |
| Batch key | Shape |
|---|---|
| rgb, points_world, ray_origins, ray_directions | [2, 8, 3, 224, 224] |
| depth, depth_valid | [2, 8, 1, 224, 224] |
| plucker | [2, 8, 6, 224, 224] |
| K | [2, 8, 3, 3] |
| T_world_from_camera | [2, 8, 4, 4] |
| original_world_from_reference | [2, 4, 4] |
| frame_numbers | [2, 8] |
| scene_scale | [2] |
A second run over 32 train samples gave a largest point reprojection error of 1.92e-4 px, a largest ray reprojection error of 1.27e-4 px, centroid distances within 6e-8 of 1, and valid depth fractions between 0.745 and 0.9997 (mean 0.943).
Every pose is multiplied on the left by the inverse of the first camera's pose, inv(Tworld←cam0) Tworld←cami, so camera 0 becomes the identity and all cameras and points are expressed in camera 0's OpenCV frame (x right, y down, z forward). The points are then unprojected in this frame and the scene scale is the distance from the origin to the mean of the valid points. Camera translations, depths and points are divided by this scale, so the valid-point centroid is at distance 1 and camera 0 stays at the origin (the centroid is not subtracted). This removes the arbitrary world frame and the metric size of each room, which a network cannot know from a few images, so the targets of every sample live in the same frame and range. The scale is between 1.9 m and 2.8 m for the three scenes above.
For pixel (u, v), K-1 [u, v, 1]T is the direction through that pixel center in camera coordinates, the point on the z = 1 plane. Only the rotation of T_world_from_camera is applied to it, because a direction does not move with the camera translation, and the result is normalized to unit length. The ray origin for every pixel of a view is the camera center, the translation column of T_world_from_camera. Since the poses are canonical, these are directions in camera 0's frame, and camera 0's directions are just its normalized camera space directions.
The Plücker coordinates of a ray are (d, m) with m = o × d. The moment is perpendicular to the plane through the world origin that contains the ray, and for a unit d its length is the distance from the origin to the line. Replacing o by any other point o + t d on the ray gives the same moment, since d × d = 0, so (d, m) describes the line itself and not a particular origin. This gives a 6 channel image per view that tells the network where every pixel's ray is, and it is continuous in both the camera position and the pixel. All rays of camera 0 pass through the origin, so its moments are zero.
Source images are mapped to [-1, 1] and concatenated with their Plücker rays (9 channels), and the target view only gets its Plücker rays (6 channels). Both are cut into 8 x 8 patches and projected to 384 dim tokens, and the same 2D sin-cos position embedding is added to every view. For each target, the tokens of all S source views and that target are joined into one sequence of (S + 1) x 784 tokens, passed through the 8 transformer blocks, and the last 784 tokens go through the RGB head and a sigmoid and are folded back into an image. The loss is the MSE between the predicted and ground truth target images.
After 2000 steps on one fixed batch (S = 2), the overfit validation PSNR is 30.94 dB (SSIM 0.871) on the 24 target images of that batch.
The chairs, table edges, whiteboard and wall colors are all reproduced. Faint 8 x 8 patch seams are still visible on flat walls.
Settings: configs/lvsm.json without changes. 14.6M parameters (patch size 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK norm). 100 epochs of 56 steps with batch size 4 and S = 2, AdamW with lr 2e-4, betas (0.9, 0.95), weight decay 0.05, 200 warmup steps, gradient clipping at 1.0 and bf16 autocast. Training took 44.6 minutes on one L40S.
At epoch 99 the train PSNR is 15.60 dB and the validation PSNR is 16.14 dB (SSIM 0.560). Validation PSNR is still rising slowly at the end (15.99 dB at epoch 79, 16.14 dB at epoch 99), so the model is not converged after 5600 steps.



Columns: source 1, source 2, ground truth, prediction. At epoch 0 the prediction is a flat gray image. By epoch 95 the dark armchair, the bright window and the floor are placed correctly as blurry blobs, but there are no edges or texture.
| Source views | Correct rays | Shuffled rays | No rays | Targets | |||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | ||
| S = 1 | 15.20 | 0.558 | 11.80 | 0.487 | 12.47 | 0.526 | 196 |
| S = 2 | 16.14 | 0.560 | 11.49 | 0.472 | 12.44 | 0.526 | 168 |
| S = 4 | 16.90 | 0.575 | 11.58 | 0.472 | 12.46 | 0.538 | 112 |
On the test split, with correct rays, PSNR is 15.51, 16.25 and 17.04 dB for S = 1, 2 and 4 (SSIM 0.538, 0.543, 0.542). Shuffled rays give 12.38, 12.23 and 11.87 dB and no rays give 12.69, 12.68 and 12.67 dB.









Columns: the S source views, ground truth, prediction.
With correct rays, PSNR goes up by 0.94 dB from S = 1 to S = 2 and by another 0.76 dB at S = 4, even though the model was only trained with S = 2. More sources cover more of the target view, so fewer target pixels have to be guessed. The targets are not the same across rows (the remaining 8 - S views of each scene are the targets), so the rows are measured on different subsets of target views. SSIM changes much less (0.558 to 0.575), because the predictions are blurry at every S and SSIM is dominated by the missing structure.
Without correct rays the model loses 3.4 to 5.3 dB. With no rays it cannot tell where the target camera is, and the MSE-optimal output is the average color of the sources, which is the flat beige image in the bottom row. With shuffled rays it confidently paints the right kind of blobs in the wrong places, which costs more under MSE than the flat average, so shuffled is about 0.6 to 1 dB below no rays. For both ablations PSNR does not change with S (11.49 to 11.80 and 12.44 to 12.47 dB), so the gain from extra source views comes entirely from using the rays to put source content at the right place in the target.
Since the baseline was still improving at epoch 100, the same configuration was also trained for 400 epochs (22400 steps, 2.56 hours of training time on one L40S, configs/lvsm_long.json). Validation PSNR reaches 15.90, 16.50, 16.92 and 17.14 dB at epochs 99, 199, 299 and 349 and stays at 17.12 dB at epoch 399, so it is close to flat over the last 50 epochs.
| 400 epochs, val | Correct rays | Shuffled rays | No rays | |||
|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| S = 1 | 15.89 | 0.588 | 11.66 | 0.482 | 12.31 | 0.525 |
| S = 2 | 17.17 | 0.597 | 11.23 | 0.461 | 12.29 | 0.519 |
| S = 4 | 18.26 | 0.616 | 11.27 | 0.457 | 12.38 | 0.527 |
On the test split the 400 epoch model gets 16.25, 17.17 and 18.39 dB with correct rays for S = 1, 2 and 4. Compared with 100 epochs, correct rays gain 0.69 to 1.36 dB, with the largest gain at S = 4, while shuffled and no rays stay at 11.2 to 12.4 dB. The longer trained model makes more use of the extra views and of the rays, so at S = 4 the gap between correct rays and no rays grows from 4.4 to 5.9 dB.


After 400 epochs the predictions show the outline of the armchair, the legs of the table and chairs and the bright window in the right places, while after 100 epochs they are colour blobs. Thin structures and textures such as the carpet pattern and the bookshelf are still blurred, which is the MSE optimum when the model is unsure about exact positions.
Question: the baseline trains with S = 2. How does training with a different number of source views, or a random number per step, change evaluation with 1, 2 and 4 source views?
Settings: four runs with the Q2.2 settings, which only differ in the training S: fixed 1, fixed 2 (the baseline), fixed 4, and S drawn uniformly from {1, 2, 3, 4} at every step (2.5 on average). Validation during training always uses S = 2, and every run is evaluated with the same protocol as Q2.2 using correct rays. Each run uses one seed.
| Training S | Eval S = 1 | Eval S = 2 | Eval S = 4 | |||
|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| 1 | 15.05 | 0.558 | 14.93 | 0.553 | 14.65 | 0.562 |
| 2 (baseline) | 15.20 | 0.558 | 16.14 | 0.560 | 16.90 | 0.575 |
| 4 | 15.23 | 0.557 | 16.33 | 0.561 | 17.25 | 0.578 |
| Random 1 to 4 | 15.21 | 0.559 | 16.12 | 0.558 | 16.91 | 0.571 |




The model trained with one source view is the only one that gets worse when it is given more views: 15.05, 14.93 and 14.65 dB for S = 1, 2 and 4. It never had to combine tokens from two source images during training, so at test time the extra views are a longer, unfamiliar sequence that spreads its attention over views it does not know how to merge. Its validation curve is also about 1.2 dB below the others for the whole run, since every training step asks it to fill the target from a single view, which teaches less about how sources map to the target.
Training with S = 4 is the best setting at every evaluation S, 0.19 dB above the baseline at S = 2 and 0.35 dB above it at S = 4, and it does not lose anything at S = 1 (15.23 against 15.20 dB). Each step has 4 x 784 source tokens, so the model sees more of the scene per update and learns to merge views, and this skill does not hurt when only one view is available. Random S lands within 0.02 dB of the baseline at all three evaluation S, which fits its average of 2.5 sources per step. In this 100 epoch setting, the number of views seen during training mattered more than matching the evaluation S.
Limitations: each setting was trained once, so differences of 0.2 to 0.3 dB between S = 2, S = 4 and random S are of the same order as seed noise and only the S = 1 result is clearly separated. All models are still improving at epoch 100, and the ranking could change with longer training. S = 4 training also uses 3920 tokens per sequence instead of 2352, so it costs more attention compute per step than the baseline.
Each view goes through the frozen DINOv2 ViT-B/14, giving 16 x 16 patch tokens at 224 x 224, which are projected to 384 dims. One camera token and four register tokens are added in front of each view, with a separate learned pair for view 0 so the model knows which camera is the reference. The 8 aggregator blocks alternate between frame attention, where the tokens of each view attend only to their own view (sequence reshaped to B V x N), and global attention, where the tokens of all views attend to each other (B x V N). The output of the last frame block and of the last global block are concatenated into 768 dim tokens. The camera head runs 4 attention blocks over the camera tokens of all views and an MLP that gives 9 numbers per view: the world-to-camera translation, the XYZW quaternion and, through a ReLU, the vertical and horizontal fields of view. The depth head maps the patch tokens back to a 224 x 224 feature map, adds a refinement from a convolution over these features and the RGB image, and outputs depth = exp(x) and confidence = 1 + exp(c).
The camera loss is the L1 error of the translation (clamped at 100) plus the L1 error of the quaternion plus 0.5 times the L1 error of the fields of view, compared directly on the 9 dim encoding. The depth loss is the trimmed mean of the absolute depth error, plus the trimmed mean of confidence x error - 0.2 log(confidence), plus a gradient loss: the absolute difference of the depth residual between neighbouring valid pixels, at strides 1, 2, 4 and 8, averaged over the four scales. The total loss is 5 x camera + depth. The depth loss becomes negative once the errors are small, because the confident pixels then get a large negative log term.
Overfitting: one training scene (036bce3393_frame_004470, 8 views), 40 epochs of 25 steps, lr 1e-3, under 4 minutes on one L40S.
| Total loss | Depth AbsRel | δ1 | Rotation error (deg) | Translation error (deg) | Pose AUC@30 | |
|---|---|---|---|---|---|---|
| Epoch 0 | 5.483 | 0.619 | 0.208 | 49.73 | 90.00 | 0.000 |
| Epoch 40 | -0.155 | 0.037 | 0.955 | 0.21 | 0.40 | 0.996 |


The predicted cameras sit on top of the ground truth frustums and the walls, tables and chairs line up with the ground truth point cloud. The predicted cloud has no holes, because depth is also predicted at the pixels where the iPhone depth is missing (black in the ground truth depth map), and these pixels add a sparse set of dark points around the edges of the scene.
Settings: configs/vggt.json without changes. Frozen DINOv2 ViT-B/14, aggregator width 384 with 6 heads and 8 blocks, 8 views per sample sampled within a range of 32 frames, 224 x 224 images, batch size 16 (14 steps per epoch), 1000 epochs. AdamW with lr 6e-4 and weight decay 0.01, 10 linear warmup epochs and then cosine decay to 0.05 x lr, loss weights 5 for camera and 1 for depth. Training took 3.67 hours of training time on one L40S (about 13 s per epoch including validation, 15.3 GB of GPU memory), split over two jobs and resumed from last.pt. best.pt is from epoch 682, which has the lowest validation loss (0.858).
The depth loss drops within the first 30 epochs and is almost flat after epoch 200. The camera loss keeps going down until about epoch 700. The validation loss is below the training loss for the whole run because they are measured on different samples: training draws new random view sets, while validation uses the fixed 8 view sets of the evaluation package.
| Depth AbsRel | δ1 | Depth RMSE (normalized) | Rotation error (deg) | Translation error (deg) | |
|---|---|---|---|---|---|
| Validation | 0.168 | 0.772 | 0.213 | 12.35 | 41.59 |
| Test | 0.181 | 0.757 | 0.225 | 9.28 | 41.60 |
| AUC | Validation | Test | ||||||
|---|---|---|---|---|---|---|---|---|
| @3 | @5 | @15 | @30 | @3 | @5 | @15 | @30 | |
| Rotation | 0.096 | 0.183 | 0.512 | 0.702 | 0.107 | 0.197 | 0.532 | 0.729 |
| Translation | 0.006 | 0.013 | 0.077 | 0.186 | 0.007 | 0.010 | 0.051 | 0.159 |
| Pose | 0.002 | 0.006 | 0.051 | 0.156 | 0.000 | 0.000 | 0.035 | 0.143 |








The room layout is right in all four scenes: the dining table and window in 09c1414f1b, the box shaped classroom in 21d970d8de and the long hallway in 280b83fcf3 line up with the ground truth. Surfaces are thicker and blurrier than in the ground truth, and depth is also filled in where the sensor has no depth (the window in 09c1414f1b and the wall on the right in 2b1dc6d6a5), which adds dark stray points.
Most of the depth error is a wrong global scale and not a wrong shape. The model has to predict depth in the canonical units, where the scale is the distance to the centroid of all valid points of all 8 views, and the depth metrics are computed without any alignment. Rescaling the predicted depth of each scene by its median ratio to the ground truth lowers AbsRel from 0.168 to 0.089 (Q3.3). The scale error is visible in 2b1dc6d6a5, where the predicted cloud is too large and its centroid is at 1.247 instead of 1.
Rotations are reasonable (mean rotation error 12.35 deg, AUC@30 0.70), but the translation directions are poor (mean error 41.6 deg, AUC@30 0.19), and the pose AUC is limited by translation. The 8 views come from the first 32 frames of an iPhone video, so many camera pairs have a short baseline compared with the scene depth, and the direction of a short baseline is hard to predict and changes a lot for a small error in position. In 09c1414f1b the predicted cameras are bunched near the first camera, and in 2b1dc6d6a5 they follow the right direction but with the wrong spacing. The model was trained on 224 scenes only, and rotation, which mostly depends on how the image content shifts, transfers to new scenes better than metric translation.
Question: how does the number of input views affect accuracy, runtime and memory, both during training and at evaluation?
Settings: two models with the Q3.2 configuration that only differ in the number of training views, 8 (the Q3.2 model) and 4 (configs/vggt_v4.json). Each best.pt is evaluated on the validation split with 2, 4, 8, 16 and 24 views from the first 32 frames, 224 px, seed 825. The reference frame is the same for every view count and the other views are drawn at random from the 32 frames. Runtime and peak memory are measured for the 8 view model with batch size 1, bf16 autocast, random input images, 3 warmup and 20 timed forward passes on one L40S.
| Eval views | Trained with 8 views | Trained with 4 views | ||||||
|---|---|---|---|---|---|---|---|---|
| AbsRel | δ1 | Rot AUC@30 | Trans AUC@30 | AbsRel | δ1 | Rot AUC@30 | Trans AUC@30 | |
| 2 | 0.122 | 0.843 | 0.731 | 0.266 | 0.109 | 0.884 | 0.671 | 0.173 |
| 4 | 0.145 | 0.819 | 0.682 | 0.158 | 0.125 | 0.867 | 0.694 | 0.167 |
| 8 | 0.168 | 0.772 | 0.702 | 0.186 | 0.149 | 0.810 | 0.691 | 0.157 |
| 16 | 0.209 | 0.695 | 0.662 | 0.168 | 0.179 | 0.765 | 0.663 | 0.131 |
| 24 | 0.212 | 0.687 | 0.659 | 0.167 | 0.184 | 0.747 | 0.669 | 0.135 |
For both models AbsRel rises steadily with the number of views, from 0.122 to 0.212 for the 8 view model. To see why, I rescaled the predicted depth of every scene by the median of ground truth / prediction over its valid pixels and recomputed AbsRel (averaged per scene, so the unaligned numbers differ slightly from the per pixel metric above).
| Eval views | Trained with 8 views | Trained with 4 views | ||||
|---|---|---|---|---|---|---|
| AbsRel | Median aligned AbsRel | Mean |log scale| | AbsRel of view 0 | AbsRel | Median aligned AbsRel | |
| 2 | 0.122 | 0.080 | 0.075 | 0.124 | 0.108 | 0.080 |
| 4 | 0.145 | 0.088 | 0.090 | 0.139 | 0.125 | 0.085 |
| 8 | 0.168 | 0.089 | 0.117 | 0.154 | 0.148 | 0.088 |
| 16 | 0.210 | 0.094 | 0.145 | 0.188 | 0.178 | 0.093 |
| 24 | 0.215 | 0.094 | 0.151 | 0.193 | 0.185 | 0.092 |
After alignment, AbsRel only goes from 0.080 to 0.094 between 2 and 24 views, while the scale error doubles from about 7.8% to 16% (exp of the mean |log scale|). The depth of view 0 also gets worse (0.124 to 0.193), even though its image is the same at every view count, so the extra views change the shared scale and not the per view shape. The canonical scale is the centroid distance over all views, and adding views from further along the video moves the centroid. With more than 8 views the model sees more views than in training and has to guess this scale for a set it was never trained on.
The 4 view model has lower raw AbsRel at every view count (0.149 against 0.168 at 8 views), but after alignment the two models are within 0.003 of each other, so the difference is in scale, not shape. The 8 view model is better at translation once there are at least 8 views (AUC@30 0.186 against 0.157 at 8 views and 0.167 against 0.135 at 24), so training with more views helps the camera head, which needs to place all cameras relative to each other. Rotation AUC@30 stays between 0.66 and 0.73 for both models with no clear trend.
| Views | 2 | 8 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| ms per forward | 33.7 | 33.7 | 38.0 | 82.0 | 211.5 | 578.4 |
| ms per view | 16.9 | 4.2 | 1.19 | 1.28 | 1.65 | 2.26 |
| Peak memory (GB) | 0.74 | 0.80 | 1.02 | 1.32 | 1.91 | 3.11 |
Up to 32 views a forward pass takes about 34 ms regardless of the number of views, because at batch 1 the GPU is not saturated and the time is dominated by fixed per kernel overhead. Beyond that the global attention dominates: each view adds 261 tokens (256 patches, a camera token and 4 registers) and global attention is quadratic in the total, so the time per view grows from 1.19 ms at 32 views to 2.26 ms at 256. Memory grows linearly, about 9 MB per extra view, since attention uses F.scaled_dot_product_attention, which does not store the full attention matrix. For training, halving the views halves the cost: 6.3 s per epoch and 1.83 hours in total for the 4 view model against 13.0 s and 3.67 hours for the 8 view model.
Each setting was trained once, and each best.pt was selected by validation loss with its own number of views (epoch 682 for 8 views, 921 for 4 views), so differences of about 0.02 in AUC are within the noise between checkpoints of one run. With 2 views there is only one camera pair per scene (28 pairs in total), so the 2 view pose AUC is noisy. All views come from a window of 32 frames, so 24 views cover most of the window and the experiment says nothing about wide baseline view sets. The benchmark uses random images and excludes data loading.