Assignment 4: Multi-View Transformers

Contents

1. Multi-View Data Preprocessing (20 points)

After unprojection and after canonicalization

After unprojection (world frame, meters, z up)After canonicalization (camera 0 frame, normalized units)
Scene 079a326597 after unprojectionScene 079a326597 after canonicalization

Scene 079a326597

Both steps put the scene where it should be. Unprojection merges the 8 views in the ScanNet++ world frame, and canonicalization moves them to camera 0's frame at unit scale. The first camera goes from (3.76, 2.08, 1.53) m to (0, 0, 0), and the valid-point centroid distance becomes 1 (scene scale 2.080 m). So the scene is in the frame the models expect.

Interactive viewer (open in new tab).

check_data.py output

split train | dataset_samples 224 | checked_samples 4
e050c15a8d_frame_007080 | valid_depth 0.978 | max_reproj_px 1.10e-04 | max_ray_reproj_px 9.41e-05 | centroid 1.0000
5f99900f09_frame_002820 | valid_depth 0.990 | max_reproj_px 1.11e-04 | max_ray_reproj_px 1.11e-04 | centroid 1.0000
47b37eb6f9_frame_002720 | valid_depth 0.961 | max_reproj_px 9.35e-05 | max_ray_reproj_px 8.90e-05 | centroid 1.0000
4c5c60fa76_frame_006690 | valid_depth 0.752 | max_reproj_px 1.07e-04 | max_ray_reproj_px 9.42e-05 | centroid 1.0000
batch shapes: rgb [2, 8, 3, 224, 224], depth [2, 8, 1, 224, 224], depth_valid [2, 8, 1, 224, 224], K [2, 8, 3, 3],
  T_world_from_camera [2, 8, 4, 4], points_world [2, 8, 3, 224, 224], ray_origins [2, 8, 3, 224, 224],
  ray_directions [2, 8, 3, 224, 224], plucker [2, 8, 6, 224, 224], frame_numbers [2, 8],
  original_world_from_reference [2, 4, 4], scene_scale [2]

Explanations

Canonicalization. All poses are expressed relative to camera 0, T_i' = T_0^-1 T_i, so camera 0 is the identity. Translations, depths and points are then divided by the distance from camera 0 to the valid-point centroid, so that distance is 1. This removes the arbitrary world frame and scale of each scene.

World-space ray directions. For pixel (u, v), d = normalize(R K^-1 [u, v, 1]), with R the camera-to-world rotation. No translation is applied since d is a direction; the ray origin is the camera center.

Plücker moment. m = o x d. It is the same for any point on the ray, since (o + s d) x d = o x d, so (d, m) encodes the line itself. With unit d, |m| is the line's distance from the origin.

2. Novel-View Synthesis with Mini-LVSM (40 points)

2.1 Overfitting (10 points)

After 2000 steps the final PSNR is 33.97 dB (SSIM 0.914).

val_epoch000.png (columns: source 1, source 2, ground truth, prediction)
LVSM overfit visualization

2.2 Training and Evaluation (30 points)

Training curves
LVSM training curves

Training improves steadily but does not converge in 100 epochs. The curve flattens near the end probably because the cosine learning rate decays. Val PSNR rises from 12.3 to 16.1 dB, and the same config trained for 300 epochs (about 3 hours) gets 15.68 / 16.92 / 17.92 dB at S = 1 / 2 / 4. So the model is underfit; the rest of this section uses the 100-epoch model.

Settings. I used the base configs/lvsm.json but decreased batch size to 2 so I could train on my e-gpu. That gives 112 steps per epoch and 11,200 steps in total, and it took about 1 hour.

Validationcorrect raysshuffled raysno rays
S = 115.17 / 0.55711.76 / 0.48612.46 / 0.527
S = 216.09 / 0.56011.45 / 0.47312.44 / 0.527
S = 416.69 / 0.57511.60 / 0.47712.48 / 0.539

PSNR (dB) / SSIM on the validation split, best.pt (epoch 95).

Training visualizations (validation batch during training): epoch 0, 50, 90
Epoch 0 Epoch 50 Epoch 90
S = 1, correctS = 2, correctS = 4, correct
Evaluation S=1Evaluation S=2Evaluation S=4
S = 2, shuffledS = 2, no rays
Evaluation S=2 shuffledEvaluation S=2 no rays

Evaluation visualizations. Columns: source views, ground truth, prediction.

Number of source views. More source views improve PSNR. More views likely show more of each target, so the model guesses less. PSNR is 15.2 dB at S = 1, 16.1 at S = 2 and 16.7 at S = 4, even though I trained only with S = 2. So the model benefits from view counts it never saw in training.

Ray conditioning. The target rays tell the model which view to render. Without them it can't know the viewpoint and outputs a blurry average of the scene colors. Correct rays give 2.7 to 4.2 dB more than no rays (about 12.5 dB), and shuffled rays are worse still (about 11.5 dB), likely because under MSE a structured guess at the wrong viewpoint costs more than a flat average. So pose conditioning matters more than the number of source views.

The predictions are blocky even with correct rays. The 8x8 patches, the small model and the MSE loss, which averages over uncertain details, are the likely causes. You can see the layout of the room if you squint but there is no fine texture, and targets that see little of the sources are blurrier. 2.3 tests the loss part.

2.3 Bonus: Loss function (10 points)

Question. Does adding a perceptual loss to the MSE make the blocky 2.2 outputs sharper, and what does it cost in PSNR and SSIM?

Settings. I follow the same setup as 2.2, with the loss MSE + w * perceptual (configs/lvsm_perceptual.json, configs/lvsm_perceptual_w01.json). I used LVSM's VGG-19 feature loss. I tried w = 0.5, the paper's value for scenes, and w = 0.1, which makes the two terms about equal.

Training curves (loss is the MSE term only)
LVSM training curves for each loss
Validation, correct raysS = 1S = 2S = 4Sharpness (S = 2)Border ratio (S = 2)LPIPS Alex / VGG (S = 2)
MSE (2.2)15.17 / 0.55716.09 / 0.56016.69 / 0.5750.2910.80.677 / 0.614
MSE + 0.1 perceptual15.17 / 0.56216.06 / 0.56216.68 / 0.5710.541.830.661 / 0.628
MSE + 0.5 perceptual14.93 / 0.55915.65 / 0.55916.05 / 0.5680.501.610.687 / 0.648

PSNR (dB) / SSIM on the validation split. Sharpness is the mean image gradient relative to GT (GT = 1). Border ratio is the gradient across the 8 px patch borders over the gradient inside patches (GT = 1.01), so a large value means a visible grid. LPIPS: lower is better.

Same validation targets, S = 2, correct rays
Ground truth and predictions for each loss

Results and conclusions. At w = 0.1 the perceptual term removes most of the patch grid without losing PSNR or SSIM. VGG features likely penalize grid patterns that MSE barely notices. At S = 2 the border ratio falls from 10.8 to 1.83 and sharpness rises from 0.29 to 0.54, while PSNR stays within 0.03 dB and SSIM within 0.006 at every S. So w = 0.1 is a clear improvement in blockiness.

A larger weight does not seem to help. Too much perceptual weight likely takes away from pixel accuracy. w = 0.5 loses 0.44 dB at S = 2 and 0.63 dB at S = 4 and is no sharper (0.50). So the paper's weight is worse than w = 0.1 for this small model.

The term reduces blockiness but not blur. The blur is likely due to the small, underfit model and to details the sources do not determine. No new texture appears, sharpness stays near half of GT, and LPIPS shows no clear gain (AlexNet 0.677 to 0.661, VGG 0.614 to 0.628). So the loss alone does not fix the 2.2 outputs.

Limitations. One run per weight and only two weights. Sharpness measures edge strength, not correctness. A faint grid remains (border ratio above 1.01).

3. Multi-View Geometry with Mini-VGGT (40 points)

3.1 Overfitting (10 points)

Overfit loss curve
VGGT overfit loss curve
Ground truthPrediction
Overfit scene, ground truthOverfit scene, prediction

The model fits a single clip almost exactly. With one clip there is nothing to generalize to, so errors should end near zero. After 40 epochs of 25 steps (default overfit settings) the total loss drops from 5.5 to below zero, AbsRel is 0.036, rotation error is 0.24 degrees and pose AUC@30 is 0.99, and the predicted cameras sit on the GT ones. So it can learn both cameras and depth.

Interactive viewer (open in new tab). Switch between ground truth and prediction in the second menu.

3.2 Training and Evaluation (30 points)

Settings. I used configs/vggt.json but decreased the batch size to 4 because of the same memory limit on my e-gpu as earlier. I kept 8 views and sampling range 32. I kept the learning rate and the per-epoch schedule, so training ran 56 steps per epoch (56,000 steps in total) and took roughly 6 hours.

Training loss_curve.png (total, camera, depth)
VGGT training loss curve

Depth is learned early and camera pose late. Depth gets most of its gain right away, while the cameras keep improving for hundreds of epochs. The val depth loss drops from 3.05 to 0.35 in 50 epochs and reaches 0.28 by epoch 1000, while the val camera loss goes from 0.18 at epoch 50 to 0.13 at 500 and flattens near 0.12 from about epoch 700. So the long training likely helps the cameras most.

SplitAbsReldelta1Rot AUC 5 / 15 / 30Trans AUC 5 / 15 / 30Pose AUC 5 / 15 / 30Rot / trans error (deg)
val0.1410.8580.169 / 0.486 / 0.6740.019 / 0.075 / 0.1710.006 / 0.062 / 0.15213.1 / 43.1
test0.1580.8100.161 / 0.456 / 0.6780.016 / 0.076 / 0.1740.003 / 0.044 / 0.15010.8 / 40.1

28 val and 28 test scenes, 8 views each, best.pt. The test set was evaluated once.

Ground truthPrediction
Scene 09c1414f1b ground truthScene 09c1414f1b prediction
Scene 21d970d8de ground truthScene 21d970d8de prediction
Scene 280b83fcf3 ground truthScene 280b83fcf3 prediction
Scene 2b1dc6d6a5 ground truthScene 2b1dc6d6a5 prediction

The first four validation scenes (09c1414f1b, 21d970d8de, 280b83fcf3, 2b1dc6d6a5).

Interactive viewer (open in a new tab). Switch between ground truth and prediction in the second menu.

Errors on held-out scenes. Depth generalizes fairly well. The depth maps are smooth, so coarse structure survives but small objects lose their shape. AbsRel is 0.14 on val and 0.16 on test (delta1 0.86 and 0.81), and all four scenes have the right room layout, but the table and chairs in the first scene merge into one blob. So depth errors are mostly in fine detail.

Cameras are the weak part, mainly translation. Pose AUC takes the worse of rotation and translation, so translation sets it. Rotation AUC@30 is about 0.68 (mean error 11 to 13 degrees), translation about 0.17 (40 to 43 degrees) and pose 0.15, and in the first scene the predicted cameras collapse into one cluster while GT camera 1 sits apart. This is consistent with the model underestimating the baselines.

3.3 Bonus: Point-map prediction and number of views (10 points)

Question. Does a direct point-map head avoid the camera error found in 3.2, and does training it change the camera and depth results?

Settings. Same as 3.2, plus a point head built like the depth head, with a VGGT-style confidence-weighted point loss plus a gradient term at weight 1. Training took roughly 7 hours. All results are on val.

Mean point error (m)3.2 modelWith point head
Predicted cameras + predicted depth0.5500.500
Direct point head0.493
Predicted cameras + GT depth0.4510.399
GT cameras + predicted depth0.3070.276
AbsReldelta1Rot AUC@30Trans AUC@30Pose AUC@30Rot / trans error (deg)
3.2 model0.1410.8580.6740.1710.15213.1 / 43.1
With point head0.1270.8620.7470.1910.17910.6 / 41.1
Validation curves
Validation curves with and without the point head
Top-down view of the point clouds (x, z in the camera 0 frame, same axes per row, point-head model)
Ground truth, unprojected and direct point clouds

Results and conclusions. The direct head does not avoid the camera error. Points in camera 0's frame likely still need the relative pose, so the head faces the same problem as the cameras. It gets 0.493 m against 0.500 m for the same model's unprojected points, while GT cameras give 0.276 m. So predicting points directly is not a way around the weak cameras.

Training the head seems to help the other tasks. The point loss likely adds geometry supervision to the shared layers. AbsRel improves from 0.141 to 0.127 and rotation AUC@30 from 0.674 to 0.747. Averaged over epochs 950 to 1000, to rule out the later checkpoint (epoch 997 against 833), AbsRel is 0.130 against 0.151, rotation AUC@30 is 0.742 against 0.675 and translation AUC@30 is 0.186 against 0.175. So the head is more useful as an extra loss than as a point output.

Limitations. One run per model and one point weight, so part of the gain could be seed noise. The direct points show horizontal banding, mostly in far regions for some reason.

Second exploration: number of views

Question. How do accuracy, runtime and memory change with 2, 4, 8 or 16 views at evaluation time, using the 8-view model?

Settings. The 3.2 model on the full val split with the 3.2 settings, changing only V. Runtime is the forward pass per scene.

VAbsReldelta1Rot AUC@30Trans AUC@30Pose AUC@30Pose pairsms / scenePeak GB
20.1230.8430.7040.1520.13628130.76
40.1340.8560.6850.1660.146168200.85
80.1410.8580.6740.1710.152784331.01
160.1590.8140.6550.1680.1393360621.35
Accuracy and cost against the number of views
VGGT accuracy and cost against number of views

Results and conclusions. Using more views than in training does not seem to help. All views come from the same 32 frames, so 16 views likely adds near-duplicates more than new information. 16 views is worse than 8 on every metric (AbsRel 0.159 against 0.141, delta1 0.814 against 0.858). So the model gains nothing past its training view count.

Cost grows about linearly with V. At this size the per-frame parts (DINO, frame attention, heads) likely dominate over the quadratic global attention. Runtime goes from 13 to 62 ms per scene and memory from 0.76 to 1.35 GB. So cost is not what limits the view count here.

Limitations. One checkpoint and one seed. Camera metrics average over 1 pair per scene at V = 2 and 120 at V = 16, so the V = 2 values are noisier.