Assignment 4 - Chan WangSing

Everything was run on one RTX 5090.

1. Multi-View Data Preprocessing

1.1 Resizing the Intrinsics

The images are cropped to a square and resized from 300 to 224 pixels, so K is scaled by s = 224/300. The focal lengths are just multiplied by s. The principal point is a position, and pixel centres sit at whole numbers, so it is scaled from the image edge instead: c' = s (c + 0.5) - 0.5.

1.2 Unprojection

Each pixel (u, v) becomes a ray K-1 [u, v, 1] with z = 1, and multiplying it by the depth gives the 3D point in camera coordinates. The camera pose (rotation R and position t) then moves it to world coordinates, X = R X_cam + t. The points from all 8 views line up into one room.

1.3 Canonicalization

Canonicalization puts every sample in the coordinate system of its first camera. All poses are multiplied by the inverse of camera 0's pose, so camera 0 sits at the origin with no rotation. Then the scene is scaled so that the centroid of the valid points is 1 unit from camera 0: camera positions, depths and points are divided by that distance, while rotations stay the same. This removes the arbitrary origin and size of each scan, so the model always sees a scene from the same starting camera and at a similar scale. The screenshots show the first camera at (0, 0, 0) and a centroid distance of 1.000.

1.4 Camera Rays and 1.5 Plücker Coordinates

World-space ray directions. Every ray starts at the camera centre. Its direction is K-1 [u, v, 1] rotated into world coordinates with R and normalised to length 1. Only the rotation is used, since a direction doesn't change when the camera moves. With all directions in the same world frame, the model knows where each pixel of each view is looking.

Plücker moment. Each ray is stored as six numbers: its direction d and its moment m = o × d. The moment is the same wherever the origin o is placed along the ray, so [d, m] describes the line itself rather than one point on it. Its length is the distance from camera 0 to the line, so all rays of camera 0 have m = 0.

Data Check

Output of the data check on 4 training samples. All checks pass, with reprojection errors around 0.0001 pixels (the tolerance is 0.01) and a centroid distance of 1.

SampleValid depthMax reprojection error (px)Max ray reprojection error (px)Centroid distance
e050c15a8d_frame_0070800.9781.11e-41.08e-41.0
5f99900f09_frame_0028200.9901.23e-41.11e-41.0
47b37eb6f9_frame_0027200.9619.78e-58.90e-51.0
4c5c60fa76_frame_0066900.7521.12e-41.02e-41.0

2. Novel-View Synthesis with Mini-LVSM

2.0 Implementation

patchify reshapes (..., C, H, W) to (L, C, H/p, p, W/p, p), permutes it to (L, H/p, W/p, p, p, C) and flattens it to (..., N, p·p·C); unpatchify does the reverse. The source RGB is shifted to [-1, 1] and concatenated with the source Plücker rays (9 channels), patchified with p = 8 (784 tokens per view) and projected to 384-D. The target rays (6 channels) have their own projection. The same 2D sine-cosine embedding is added to both, and every target gets its own sequence: all source tokens followed by its 784 query tokens, with the targets folded into the batch so they never see each other. After the input norm and the 8 transformer blocks, the last 784 tokens go through the output norm, the RGB head and a sigmoid, and are unpatchified. The loss is the MSE over all target pixels. Training runs under bfloat16 autocast with the loss computed in float32.

2.1 Overfitting

After 2000 steps on one fixed batch the PSNR is 30.89 dB (SSIM 0.870). Columns: the two source views, the ground truth and the prediction. The predictions are slightly blurry with faint 8x8 patch edges. Rows 2 and 3 show a blue wall and a red chair that are in neither source view, which the model can only draw because it memorised this batch.

2.2 Training and Evaluation

Settings: patch size 8, width 384, 8 layers, 64-D heads with QK norm (14.6M parameters); 2 source and 6 target views per sample at 224x224, batch size 4, 100 epochs (5600 steps); AdamW with learning rate 2e-4, betas (0.9, 0.95) and weight decay 0.05, 200 warmup steps then cosine decay to 10%, gradient clipping 1.0, bfloat16. Training took 17 minutes. The best validation PSNR was at the last epoch, 16.14 dB with SSIM 0.560.

Train and validation PSNR stay close (15.6 and 16.1 dB at the end) and are both still rising, so the model is under-trained rather than overfitting. Validation pictures during training, epochs 10 and 95 (columns: sources, ground truth, prediction):

epoch 10
epoch 95

Evaluation of the best checkpoint on the validation split:

Source viewsTarget raysPSNR (dB)SSIMImages
1correct15.200.558196
1shuffled11.810.487196
1none12.460.526196
2correct16.140.560168
2shuffled11.500.472168
2none12.440.525168
4correct16.880.576112
4shuffled11.580.473112
4none12.460.536112
1 source view, correct rays
2 source views, correct rays
4 source views, correct rays
2 source views, shuffled rays
2 source views, no rays

Source views. PSNR goes from 15.20 to 16.14 to 16.88 dB with 1, 2 and 4 source views, even though the model was trained with 2. Every extra view shows more of the room and the transformer just gets a longer sequence. The image counts differ because 8 views minus S sources leave 7, 6 or 4 targets per scene, so the rows are not computed on exactly the same target images.

Ray conditioning. With zero target rays every target of a scene gets the same flat average colour and the PSNR stays at 12.4 to 12.5 dB however many source views there are: without the camera the model can't tell which part of the room to draw. With shuffled rays it draws the view of the neighbouring target (each row of the shuffled image matches the row above it in the correct one), and the PSNR drops to 11.5 to 11.8 dB, below no rays at all, since a confident wrong view costs more under MSE than a flat average. Correct rays are 3 to 4.4 dB better than no rays, so the model does use the target camera.

The predictions are still blurry blobs with the right layout and colours. With 224 training scenes, a small model and 17 minutes of training, it can't infer fine detail, and under MSE an uncertain model predicts the average, which looks blurry. Parts of the target that aren't visible in any source view are the worst.

2.3 Bonus: SSIM Loss

Question. Can an SSIM term make the blurry predictions sharper, and what does it cost in PSNR?

Setup. Same config, seed and 100 epochs as Q2.2; only the training loss changes to MSE + 0.2 (1 - SSIM). Validation still reports plain MSE, and the best checkpoint is chosen by validation PSNR in both runs.

Source viewsPSNR, MSEPSNR, MSE + SSIMSSIM, MSESSIM, MSE + SSIM
115.2014.800.5580.577
216.1415.450.5600.580
416.8815.810.5760.594
ground truth, MSE, MSE + SSIM (2 source views)

Results. SSIM improves by about 0.02 for every number of source views and PSNR drops by 0.4 to 1.1 dB. That trade-off is expected, since PSNR is just MSE on a log scale and the model no longer optimises MSE alone. Visually, the SSIM run gets rid of the 8x8 patch seams and the blobs are smooth, but they aren't sharper: no chair legs or shelves appear. To check this I measured the mean absolute difference between neighbouring pixels of the 4 predictions on the left: 11.6 for the ground truth, 3.2 for MSE and 2.1 for MSE + SSIM. Most of the "detail" in the MSE predictions was the false edges between patches.

Conclusion. SSIM compares the local mean, contrast and structure in small windows, and the patch seams are structure that isn't in the real image, so the SSIM term removes them. It can't add detail the model doesn't know, though: the network is still unsure about parts of the room it can't see and predicts a smooth average. Limitations: one weight (0.2), one run per setting without repeated seeds, and short training. A perceptual loss or a larger, longer-trained model would be the next things to try.

3. Multi-View Geometry with Mini-VGGT

3.0 Implementation

The frozen DINOv2 ViT-B/14 features of each view (256 patches) are projected to 384-D and prefixed with one camera token and four register tokens, with separate learned tokens for camera 0. The aggregator alternates 16 blocks: even blocks attend within each view on (B·V, N, D) and odd blocks attend across all views on (B, V·N, D), with the 2D positions repeated for every view and no view index. The outputs of the last frame block and the last global block are concatenated (768-D) and normalised. The camera head turns the camera token into 9 numbers, a world-to-camera translation, an XYZW quaternion and two fields of view, with a ReLU on the fields of view only. The depth head maps each patch token to a 14x14 depth patch, adds a residual from a small convolution over the depth and the RGB image, and outputs exp(depth) and 1 + exp(confidence).

The camera loss is the L1 distance between the predicted and ground-truth encodings (translation errors clamped at 100, fields of view weighted by 0.5). The depth loss is the trimmed mean of the absolute error, plus the confidence term conf · error - 0.2 log(conf), plus an L1 loss on the horizontal and vertical differences of the depth residual at strides 1, 2, 4 and 8. The total is 5 x camera + 1 x depth.

3.1 Overfitting

One fixed training clip, 40 epochs of 25 steps with learning rate 1e-3 (about 1.5 minutes). The total loss goes below zero because of the -0.2 log(conf) term once the confidence grows.

Epoch 0Epoch 40
Depth AbsRel0.6190.037
Depth δ10.2080.955
Mean rotation error49.7°0.14°
Mean translation direction error90.0°0.56°
Joint pose AUC@300.000.99
ground truth
predicted cameras and depth

The predicted cameras and points almost exactly match the ground truth of the memorised clip.

3.2 Training and Evaluation

Settings (the defaults): 8 frame/global pairs, width 384, 6 heads, frozen DINOv2 ViT-B/14; 8 views per sample with sampling range 32 at 224 px, batch size 16 (14 steps per epoch), 1000 epochs; AdamW with learning rate 6e-4 and weight decay 0.01, 10 warmup epochs and cosine decay to 3e-5, gradient clipping 1.0, bfloat16. Training took 1.9 hours. The best checkpoint, with the lowest validation loss, is from epoch 967.

The validation loss is below the training loss for the whole run but follows the same trend, and it was still slowly falling at epoch 1000. The depth loss drops within the first 20 epochs and then flattens, while the camera loss keeps decreasing until the end.

MetricValidationTest
Depth AbsRel (lower is better)0.1490.162
Depth δ10.8220.783
Rotation AUC @5 / @15 / @300.187 / 0.490 / 0.6820.199 / 0.518 / 0.712
Translation AUC @5 / @15 / @300.016 / 0.087 / 0.1950.009 / 0.048 / 0.159
Joint pose AUC @5 / @15 / @300.005 / 0.067 / 0.1710.002 / 0.028 / 0.141

Four validation scenes, ground truth on the left and prediction on the right:

Depth is the strongest part: about 15% relative error, with around 80% of the pixels within 25% of the ground truth on unseen scenes. The DINOv2 features already carry a lot of depth information, which also explains how fast the depth loss drops. In the point clouds the floor, walls and overall layout are recovered, but surfaces are thicker and noisier than in the ground truth, small objects such as chair legs and table edges are smeared, and the 14x14 patch structure shows in the predicted depth maps.

Rotation is roughly right but not precise: the AUC is about 0.7 at 30° but only about 0.2 at 5°. Translation is the weakest part and limits the joint pose AUC, which takes the worse of the two errors per pair. The frames come from one handheld sequence, so cameras are often only a few centimetres apart and the direction between them is very sensitive: with a 10 cm baseline, a 5 cm error is already a 27° direction error. Rotation errors don't depend on the baseline. The predicted cameras in the screenshots are close to the ground truth ones, but at these distances small offsets are enough to give large direction errors.

Validation and test are close, so the model behaves consistently on two different sets of unseen scenes. Test is slightly worse for depth and translation and slightly better for rotation. The camera loss was still going down at the end, so longer training should help the cameras most.

3.3 Bonus: Number of Views

Question. The model was trained with 8 views. How do accuracy, runtime and memory change when it gets 2, 4, 8, 16 or 32 views at evaluation time?

Setup. I evaluated the Q3.2 model on val and test with the same settings as Q3.2, changing only the number of views. With 8 views it reproduces the Q3.2 numbers exactly. Runtime and peak GPU memory are for a forward pass on a batch of 16 scenes, averaged over 10 runs.

ViewsAbsRelδ1Rotation AUC@30Translation AUC@30Pose AUC@30RuntimeMemory
20.122 / 0.1200.846 / 0.8550.677 / 0.7260.104 / 0.1920.100 / 0.14424 ms1.1 GB
40.139 / 0.1450.833 / 0.8200.686 / 0.7290.170 / 0.1580.147 / 0.14043 ms1.4 GB
80.149 / 0.1620.822 / 0.7830.682 / 0.7120.195 / 0.1590.171 / 0.14193 ms2.1 GB
160.175 / 0.1740.798 / 0.7700.669 / 0.7080.160 / 0.1620.138 / 0.144209 ms3.4 GB
320.179 / 0.1740.779 / 0.7720.651 / 0.7180.154 / 0.1650.130 / 0.147453 ms6.2 GB

Metrics are validation / test.

validation (top) and test (bottom)

Results. Depth gets worse as views are added, on both splits (AbsRel 0.12 to 0.18). The depth is predicted in canonical units where the centroid of the points of all views is at distance 1, so with more views the scale of each view depends on more frames. 16 and 32 views also give global-attention sequences two to four times longer than in training. The cameras don't benefit from extra views either: rotation AUC stays between 0.65 and 0.73, and translation AUC peaks at the training count of 8 on val and is flat on test. The model doesn't learn to use more views than it was trained with. The cost grows quickly: runtime roughly doubles with every doubling of views, a bit more than that at 16 and 32 since global attention is quadratic in the number of tokens, and memory goes from 1.1 to 6.2 GB.

Limitations. Views are only changed at evaluation time; training with a random number of views could let the model use extra views. With 2 views there is only one camera pair per scene (28 pairs in total, against 13,888 with 32 views), so the 2-view camera AUCs are noisy.