Everything was run on one RTX 5090.
The images are cropped to a square and resized from 300 to 224 pixels, so K is scaled by s = 224/300. The
focal lengths are just multiplied by s. The principal point is a position, and pixel centres sit at whole
numbers, so it is scaled from the image edge instead: c' = s (c + 0.5) - 0.5.
Each pixel (u, v) becomes a ray K-1 [u, v, 1] with z = 1, and multiplying it by the
depth gives the 3D point in camera coordinates. The camera pose (rotation R and position t) then moves it to
world coordinates, X = R X_cam + t. The points from all 8 views line up into one room.



Canonicalization puts every sample in the coordinate system of its first camera. All poses are multiplied by the inverse of camera 0's pose, so camera 0 sits at the origin with no rotation. Then the scene is scaled so that the centroid of the valid points is 1 unit from camera 0: camera positions, depths and points are divided by that distance, while rotations stay the same. This removes the arbitrary origin and size of each scan, so the model always sees a scene from the same starting camera and at a similar scale. The screenshots show the first camera at (0, 0, 0) and a centroid distance of 1.000.


World-space ray directions. Every ray starts at the camera centre. Its direction is
K-1 [u, v, 1] rotated into world coordinates with R and normalised to length 1. Only
the rotation is used, since a direction doesn't change when the camera moves. With all directions in the same
world frame, the model knows where each pixel of each view is looking.
Plücker moment. Each ray is stored as six numbers: its direction d and its moment
m = o × d. The moment is the same wherever the origin o is placed along the ray, so
[d, m] describes the line itself rather than one point on it. Its length is the distance from camera 0 to the
line, so all rays of camera 0 have m = 0.
Output of the data check on 4 training samples. All checks pass, with reprojection errors around 0.0001 pixels (the tolerance is 0.01) and a centroid distance of 1.
| Sample | Valid depth | Max reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|
| e050c15a8d_frame_007080 | 0.978 | 1.11e-4 | 1.08e-4 | 1.0 |
| 5f99900f09_frame_002820 | 0.990 | 1.23e-4 | 1.11e-4 | 1.0 |
| 47b37eb6f9_frame_002720 | 0.961 | 9.78e-5 | 8.90e-5 | 1.0 |
| 4c5c60fa76_frame_006690 | 0.752 | 1.12e-4 | 1.02e-4 | 1.0 |
patchify reshapes (..., C, H, W) to (L, C, H/p, p, W/p, p), permutes it to
(L, H/p, W/p, p, p, C) and flattens it to (..., N, p·p·C); unpatchify does the reverse. The
source RGB is shifted to [-1, 1] and concatenated with the source Plücker rays (9 channels), patchified with
p = 8 (784 tokens per view) and projected to 384-D. The target rays (6 channels) have their own projection.
The same 2D sine-cosine embedding is added to both, and every target gets its own sequence: all source
tokens followed by its 784 query tokens, with the targets folded into the batch so they never see each
other. After the input norm and the 8 transformer blocks, the last 784 tokens go through the output norm,
the RGB head and a sigmoid, and are unpatchified. The loss is the MSE over all target pixels. Training runs
under bfloat16 autocast with the loss computed in float32.
After 2000 steps on one fixed batch the PSNR is 30.89 dB (SSIM 0.870). Columns: the two source views, the ground truth and the prediction. The predictions are slightly blurry with faint 8x8 patch edges. Rows 2 and 3 show a blue wall and a red chair that are in neither source view, which the model can only draw because it memorised this batch.
Settings: patch size 8, width 384, 8 layers, 64-D heads with QK norm (14.6M parameters); 2 source and 6 target views per sample at 224x224, batch size 4, 100 epochs (5600 steps); AdamW with learning rate 2e-4, betas (0.9, 0.95) and weight decay 0.05, 200 warmup steps then cosine decay to 10%, gradient clipping 1.0, bfloat16. Training took 17 minutes. The best validation PSNR was at the last epoch, 16.14 dB with SSIM 0.560.
Train and validation PSNR stay close (15.6 and 16.1 dB at the end) and are both still rising, so the model is under-trained rather than overfitting. Validation pictures during training, epochs 10 and 95 (columns: sources, ground truth, prediction):


Evaluation of the best checkpoint on the validation split:
| Source views | Target rays | PSNR (dB) | SSIM | Images |
|---|---|---|---|---|
| 1 | correct | 15.20 | 0.558 | 196 |
| 1 | shuffled | 11.81 | 0.487 | 196 |
| 1 | none | 12.46 | 0.526 | 196 |
| 2 | correct | 16.14 | 0.560 | 168 |
| 2 | shuffled | 11.50 | 0.472 | 168 |
| 2 | none | 12.44 | 0.525 | 168 |
| 4 | correct | 16.88 | 0.576 | 112 |
| 4 | shuffled | 11.58 | 0.473 | 112 |
| 4 | none | 12.46 | 0.536 | 112 |





Source views. PSNR goes from 15.20 to 16.14 to 16.88 dB with 1, 2 and 4 source views, even though the model was trained with 2. Every extra view shows more of the room and the transformer just gets a longer sequence. The image counts differ because 8 views minus S sources leave 7, 6 or 4 targets per scene, so the rows are not computed on exactly the same target images.
Ray conditioning. With zero target rays every target of a scene gets the same flat average colour and the PSNR stays at 12.4 to 12.5 dB however many source views there are: without the camera the model can't tell which part of the room to draw. With shuffled rays it draws the view of the neighbouring target (each row of the shuffled image matches the row above it in the correct one), and the PSNR drops to 11.5 to 11.8 dB, below no rays at all, since a confident wrong view costs more under MSE than a flat average. Correct rays are 3 to 4.4 dB better than no rays, so the model does use the target camera.
The predictions are still blurry blobs with the right layout and colours. With 224 training scenes, a small model and 17 minutes of training, it can't infer fine detail, and under MSE an uncertain model predicts the average, which looks blurry. Parts of the target that aren't visible in any source view are the worst.
Question. Can an SSIM term make the blurry predictions sharper, and what does it cost in PSNR?
Setup. Same config, seed and 100 epochs as Q2.2; only the training loss changes to
MSE + 0.2 (1 - SSIM). Validation still reports plain MSE, and the best checkpoint is chosen
by validation PSNR in both runs.
| Source views | PSNR, MSE | PSNR, MSE + SSIM | SSIM, MSE | SSIM, MSE + SSIM |
|---|---|---|---|---|
| 1 | 15.20 | 14.80 | 0.558 | 0.577 |
| 2 | 16.14 | 15.45 | 0.560 | 0.580 |
| 4 | 16.88 | 15.81 | 0.576 | 0.594 |

Results. SSIM improves by about 0.02 for every number of source views and PSNR drops by 0.4 to 1.1 dB. That trade-off is expected, since PSNR is just MSE on a log scale and the model no longer optimises MSE alone. Visually, the SSIM run gets rid of the 8x8 patch seams and the blobs are smooth, but they aren't sharper: no chair legs or shelves appear. To check this I measured the mean absolute difference between neighbouring pixels of the 4 predictions on the left: 11.6 for the ground truth, 3.2 for MSE and 2.1 for MSE + SSIM. Most of the "detail" in the MSE predictions was the false edges between patches.
Conclusion. SSIM compares the local mean, contrast and structure in small windows, and the patch seams are structure that isn't in the real image, so the SSIM term removes them. It can't add detail the model doesn't know, though: the network is still unsure about parts of the room it can't see and predicts a smooth average. Limitations: one weight (0.2), one run per setting without repeated seeds, and short training. A perceptual loss or a larger, longer-trained model would be the next things to try.
The frozen DINOv2 ViT-B/14 features of each view (256 patches) are projected to 384-D and prefixed with one
camera token and four register tokens, with separate learned tokens for camera 0. The aggregator
alternates 16 blocks: even blocks attend within each view on (B·V, N, D) and odd blocks attend across all
views on (B, V·N, D), with the 2D positions repeated for every view and no view index. The outputs of the
last frame block and the last global block are concatenated (768-D) and normalised. The camera head turns
the camera token into 9 numbers, a world-to-camera translation, an XYZW quaternion and two fields of view,
with a ReLU on the fields of view only. The depth head maps each patch token to a 14x14 depth patch, adds a
residual from a small convolution over the depth and the RGB image, and outputs exp(depth) and
1 + exp(confidence).
The camera loss is the L1 distance between the predicted and ground-truth encodings (translation errors
clamped at 100, fields of view weighted by 0.5). The depth loss is the trimmed mean of the absolute error,
plus the confidence term conf · error - 0.2 log(conf), plus an L1 loss on the horizontal and
vertical differences of the depth residual at strides 1, 2, 4 and 8. The total is 5 x camera + 1 x depth.
One fixed training clip, 40 epochs of 25 steps with learning rate 1e-3 (about 1.5 minutes). The total loss
goes below zero because of the -0.2 log(conf) term once the confidence grows.
| Epoch 0 | Epoch 40 | |
|---|---|---|
| Depth AbsRel | 0.619 | 0.037 |
| Depth δ1 | 0.208 | 0.955 |
| Mean rotation error | 49.7° | 0.14° |
| Mean translation direction error | 90.0° | 0.56° |
| Joint pose AUC@30 | 0.00 | 0.99 |


The predicted cameras and points almost exactly match the ground truth of the memorised clip.
Settings (the defaults): 8 frame/global pairs, width 384, 6 heads, frozen DINOv2 ViT-B/14; 8 views per sample with sampling range 32 at 224 px, batch size 16 (14 steps per epoch), 1000 epochs; AdamW with learning rate 6e-4 and weight decay 0.01, 10 warmup epochs and cosine decay to 3e-5, gradient clipping 1.0, bfloat16. Training took 1.9 hours. The best checkpoint, with the lowest validation loss, is from epoch 967.
The validation loss is below the training loss for the whole run but follows the same trend, and it was still slowly falling at epoch 1000. The depth loss drops within the first 20 epochs and then flattens, while the camera loss keeps decreasing until the end.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel (lower is better) | 0.149 | 0.162 |
| Depth δ1 | 0.822 | 0.783 |
| Rotation AUC @5 / @15 / @30 | 0.187 / 0.490 / 0.682 | 0.199 / 0.518 / 0.712 |
| Translation AUC @5 / @15 / @30 | 0.016 / 0.087 / 0.195 | 0.009 / 0.048 / 0.159 |
| Joint pose AUC @5 / @15 / @30 | 0.005 / 0.067 / 0.171 | 0.002 / 0.028 / 0.141 |
Four validation scenes, ground truth on the left and prediction on the right:








Depth is the strongest part: about 15% relative error, with around 80% of the pixels within 25% of the ground truth on unseen scenes. The DINOv2 features already carry a lot of depth information, which also explains how fast the depth loss drops. In the point clouds the floor, walls and overall layout are recovered, but surfaces are thicker and noisier than in the ground truth, small objects such as chair legs and table edges are smeared, and the 14x14 patch structure shows in the predicted depth maps.
Rotation is roughly right but not precise: the AUC is about 0.7 at 30° but only about 0.2 at 5°. Translation is the weakest part and limits the joint pose AUC, which takes the worse of the two errors per pair. The frames come from one handheld sequence, so cameras are often only a few centimetres apart and the direction between them is very sensitive: with a 10 cm baseline, a 5 cm error is already a 27° direction error. Rotation errors don't depend on the baseline. The predicted cameras in the screenshots are close to the ground truth ones, but at these distances small offsets are enough to give large direction errors.
Validation and test are close, so the model behaves consistently on two different sets of unseen scenes. Test is slightly worse for depth and translation and slightly better for rotation. The camera loss was still going down at the end, so longer training should help the cameras most.
Question. The model was trained with 8 views. How do accuracy, runtime and memory change when it gets 2, 4, 8, 16 or 32 views at evaluation time?
Setup. I evaluated the Q3.2 model on val and test with the same settings as Q3.2, changing only the number of views. With 8 views it reproduces the Q3.2 numbers exactly. Runtime and peak GPU memory are for a forward pass on a batch of 16 scenes, averaged over 10 runs.
| Views | AbsRel | δ1 | Rotation AUC@30 | Translation AUC@30 | Pose AUC@30 | Runtime | Memory |
|---|---|---|---|---|---|---|---|
| 2 | 0.122 / 0.120 | 0.846 / 0.855 | 0.677 / 0.726 | 0.104 / 0.192 | 0.100 / 0.144 | 24 ms | 1.1 GB |
| 4 | 0.139 / 0.145 | 0.833 / 0.820 | 0.686 / 0.729 | 0.170 / 0.158 | 0.147 / 0.140 | 43 ms | 1.4 GB |
| 8 | 0.149 / 0.162 | 0.822 / 0.783 | 0.682 / 0.712 | 0.195 / 0.159 | 0.171 / 0.141 | 93 ms | 2.1 GB |
| 16 | 0.175 / 0.174 | 0.798 / 0.770 | 0.669 / 0.708 | 0.160 / 0.162 | 0.138 / 0.144 | 209 ms | 3.4 GB |
| 32 | 0.179 / 0.174 | 0.779 / 0.772 | 0.651 / 0.718 | 0.154 / 0.165 | 0.130 / 0.147 | 453 ms | 6.2 GB |
Metrics are validation / test.
validation (top) and test (bottom)
Results. Depth gets worse as views are added, on both splits (AbsRel 0.12 to 0.18). The depth is predicted in canonical units where the centroid of the points of all views is at distance 1, so with more views the scale of each view depends on more frames. 16 and 32 views also give global-attention sequences two to four times longer than in training. The cameras don't benefit from extra views either: rotation AUC stays between 0.65 and 0.73, and translation AUC peaks at the training count of 8 on val and is flat on test. The model doesn't learn to use more views than it was trained with. The cost grows quickly: runtime roughly doubles with every doubling of views, a bit more than that at 16 and 32 since global attention is quadratic in the number of tokens, and memory goes from 1.1 to 6.2 GB.
Limitations. Views are only changed at evaluation time; training with a random number of views could let the model use extra views. With 2 views there is only one camera pair per scene (28 pairs in total, against 13,888 with 32 views), so the 2-view camera AUCs are noisy.