16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

1.1–1.2 Resized intrinsics and depth unprojection

Points and cameras in the original ScanNet++ world frame (meters), after Q1.1 and Q1.2. Red frustum is the first sampled camera, blue are the other 7 views. 5 training scenes, 10000 points per view.

unprojection scene 0
Scene 0
unprojection scene 1
Scene 1
unprojection scene 2
Scene 2
unprojection scene 3
Scene 3
unprojection scene 4
Scene 4

1.3 Canonicalization

The same frames after canonicalization. Camera 0 is now the identity (red frustum at the origin, looking down +Z) and the valid-point centroid is 1 unit from the origin. The panel on the right shows the first camera center and the centroid distance.

canonicalization scene 0
Scene 0
canonicalization scene 1
Scene 1
canonicalization scene 2
Scene 2
canonicalization scene 3
Scene 3
canonicalization scene 4
Scene 4

1.7 Final check

$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [{
      "sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9779127240180969, "max_reprojection_error_px": 0.00011009880108758807, "max_ray_reprojection_error_px": 0.00010764275066321716, "centroid_distance": 1.0
    }, {
      "sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9903116226196289, "max_reprojection_error_px": 0.00012302029062993824, "max_ray_reprojection_error_px": 0.00011108566104667261, "centroid_distance": 0.9999999403953552
    }, {
      "sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224, "valid_depth_fraction": 0.9609947800636292, "max_reprojection_error_px": 9.881925507215783e-05, "max_ray_reprojection_error_px": 8.897326188161969e-05, "centroid_distance": 1.0
    }, {
      "sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224, "valid_depth_fraction": 0.7516716122627258, "max_reprojection_error_px": 0.00011280165199423209, "max_ray_reprojection_error_px": 0.00010225349979009479, "centroid_distance": 0.9999999403953552
    }],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Explanations

Canonicalization. Every sample is re-expressed in the frame of its first sampled camera: Ti' = T0−1 Ti, so camera 0 becomes the identity and the other poses are relative to it. The depth of camera 0 is then unprojected in this frame and the distance from the origin to the centroid of all valid points, s = ||mean(p)||, is used as the scene scale. Translations, depth and points are all divided by s (the centroid is not subtracted, so camera 0 stays at the origin). The ScanNet++ world frame is arbitrary per scene and room sizes vary a lot, so without this the network would have to cope with arbitrary global rotations, translations and scales. After canonicalization every sample has the same reference frame and roughly the same scale.

World-space ray directions. For pixel (u, v) the camera-space direction is K−1[u, v, 1]T (pixel centers are integer coordinates, and K is the resized intrinsic matrix from Q1.1). It is rotated into the canonical world frame with the camera-to-world rotation, d = normalize(R K−1[u, v, 1]T), and the origin of every ray is the camera center t. So each pixel knows where it looks from and in which direction, in the same coordinate frame for all views.

Plücker moment. A ray is stored as six numbers [d, m] with m = o × d. The moment is the same for any point o on the line (o + λd gives (o + λd) × d = o × d), so the 6D vector describes the line itself and not a particular origin. It is perpendicular to d, and its norm is the distance from the world origin to the line. Together, d and m encode the camera position and orientation of every pixel densely, which is what LVSM uses as its camera conditioning.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR = 30.98 dB   (SSIM 0.872, MSE 0.00085). The average training PSNR over all 2000 steps is 25.41 dB, because it includes the start of training.

overfit visualization
val_epoch000.png. Columns: source view 1, source view 2, ground truth target, prediction

2.2 Training and evaluation

Training settings: Each sample has 8 views from a 32-frame window, S = 2 sources and T = 6 targets, at 224×224, batch size 4. 100 epochs × 56 steps = 5600 steps. AdamW (lr 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps then cosine decay to 10%, gradient clipping 1.0, bfloat16 autocast with the loss in float32. One L40S, 47 minutes. Validation uses 8 views from the first 32 prepared frames of each of the 28 validation scenes. best.pt is the last epoch (99), since validation PSNR was still going up.

LVSM training curves
Training curves (log.jsonl). Validation is a bit higher than training because training samples are random 8-view subsets of the 32-frame window, while validation uses a fixed selection.

Validation metrics of best.pt

Source viewsTarget raysPSNR (dB)SSIMTarget images
S = 1correct15.190.5580196
S = 1shuffled11.800.4872196
S = 1none12.470.5305196
S = 2correct16.130.5599168
S = 2shuffled11.490.4719168
S = 2none12.450.5285168
S = 4correct16.880.5751112
S = 4shuffled11.580.4726112
S = 4none12.460.5385112

Training visualizations (val_epoch*.png; columns: source 1, source 2, ground truth, prediction):

Epoch 0
Epoch 50
Epoch 95

Evaluation visualizations (eval_val/S*_*.png; columns: source views, ground truth, prediction):

S = 1, correct rays
S = 2, correct rays
S = 4, correct rays
S = 2, correct
S = 2, shuffled
S = 2, none (zero rays)

Quality of the baseline. After 5600 steps the model has learned the coarse layout and colours of the target view (where the window, floor, chairs and walls are), but the predictions are very blurry and every 8×8 patch is close to a flat colour. The same model reaches 31 dB when it memorizes one batch (Q2.1), so the architecture can produce detail. With 224 training scenes and a short schedule it has not yet learned to copy texture from the sources at the right place, and predicting a blurred average is the MSE-optimal answer under that uncertainty. Training and validation PSNR were both still rising at epoch 100, so the model is under-trained rather than overfitting.

Effect of the number of source views. With the true target rays, PSNR goes from 15.19 (S = 1) to 16.13 (S = 2) to 16.88 dB (S = 4), and SSIM also goes up. More sources cover more of the target view, so less of the target has to be guessed. Using S = 4 at test time also works even though the model was trained with S = 2 only. Attention works on any number of source tokens. Because the targets are the remaining 8 − S views, the target sets differ slightly between rows, but the trend is consistent.

Effect of the ray-conditioning ablation. Removing or corrupting the target rays drops PSNR by about 3.5–5 dB for every S (e.g. S = 2: 16.13 → 11.49 dB shuffled, 12.45 dB none). With none the query tokens only carry the patch position (all-zero rays never appear in training), and the model outputs an almost uniform image in the average colour of the source views, the same for every target, with no layout at all. Shuffled is the worst, slightly below none, because the model confidently renders a different view (the wrong target) instead of hedging towards the mean, and a confident wrong image costs more MSE than a blurry average. So the model uses the target Plücker rays to decide what to render, and the images change with the camera.

2.3 Bonus: number of source views during training

Question. Does the number of source views used during training change how well the model uses 1, 2 or 4 source views at test time?

Setup. Three runs with exactly the Q2.2 settings, the same seed and the same 5600 steps. Only num_source_views in the config changes, giving T = 7, 6 and 4 targets per sample. Every best.pt (epoch 99 for all three) is evaluated with S = 1, 2, 4 and correct rays. Within a column the target images are identical, so rows can be compared directly.

Val PSNR (dB) / SSIMEval S = 1Eval S = 2Eval S = 4
Train S = 115.11 / 0.55914.90 / 0.55314.64 / 0.561
Train S = 215.19 / 0.55816.13 / 0.56016.88 / 0.575
Train S = 415.29 / 0.55616.40 / 0.56317.46 / 0.582
bonus curves
Training curves of the three runs. The validation curve of each run uses its own S.

Results.

  1. The model trained with S = 1 does not get better with more sources. Its PSNR even goes down (15.11 → 14.90 → 14.64 dB). It never had to combine several images during training, and a longer sequence with more source tokens changes the attention statistics it was trained on.
  2. Training with more sources helps in every column. Train S = 4 is best for eval S = 4 (17.46 vs 16.88 dB for train S = 2) and for eval S = 2 (16.40 vs 16.13 dB), and it does not lose anything for single-view inputs (15.29 vs 15.19 vs 15.11 dB).
  3. Train S = 4 sees fewer target images per step (4 instead of 6) but longer sequences, so it is about 10% slower per epoch (29 vs 26 s) and uses more memory. The extra sources give more overlapping content to copy from, which seems to teach the model to attend to and fuse matching content across views. It learns this faster than when most of the target has to be guessed.

Limitations. One seed per setting, and the differences at eval S = 1 (0.2 dB) are within run-to-run noise. All models are under-trained (PSNR 15–17 dB), so the ranking could change with a longer schedule. A random S per step would probably give the most robust model, but it needs a change to the training loop, so I kept this to a config-only comparison.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

default model (frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads), one fixed training clip with 8 views, 40 epochs × 25 steps, lr 1e-3, L40S, about 5 minutes.

overfit loss curve
Overfit loss curve (total, camera and depth). The total and depth loss go below zero because of the −0.2 log(confidence) term once the confidence grows on well fitted pixels.
Overfit clipDepth AbsRelDepth δ1Rot. error (°)Transl. error (°)Pose AUC@5Pose AUC@30
Epoch 00.6190.20849.790.00.0000.000
Epoch 400.0360.9560.290.450.9710.995
overfit GT
Ground truth cameras and depth-unprojected points
overfit prediction
Predicted cameras + depth. First camera center (0.0004, −0.0009, 0.0004), centroid distance 0.996

The predicted cameras fall on top of the GT frustums and the point cloud has the same layout as the GT. The predicted depth is a little blockier (the dense head predicts one 14×14 patch per token and the small RGB refinement only partly sharpens the edges).

3.2 Training and evaluation

Training settings: frozen DINOv2 ViT-B/14 encoder, 8 frame/global attention pairs (width 384, 6 heads), camera head with 4 cross-view blocks, depth head with RGB refinement. 8 views per sample from a 32-frame window, 224×224, batch size 16, 1000 epochs × 14 steps = 14000 steps. AdamW (lr 6e-4, weight decay 0.01), 10 warmup epochs then cosine decay to 5% of the peak lr, gradient clipping 1.0, bfloat16 autocast. Loss = 5 × camera + 1 × depth. One NVIDIA L40S, 4.05 hours of training (about 14 s per epoch including validation). best.pt is epoch 843, the epoch with the lowest validation loss (0.804).

training loss curve
Training loss curve (total, camera and depth). Validation is lower than training because training samples are random 8-frame subsets in random order, while validation uses the fixed selection that always keeps frame 0 as the reference.

Metrics of best.pt (8 views from the first 32 prepared frames, 224px, seed 825, full splits, 28 scenes and 784 pose pairs each):

SplitDepth AbsRel ↓Depth δ1 ↑Rot AUC@5Rot AUC@15Rot AUC@30Transl AUC@5Transl AUC@15Transl AUC@30Pose AUC@5Pose AUC@15Pose AUC@30
Validation0.1450.8200.1890.5450.7360.0140.0710.1780.0060.0600.169
Test0.1490.8270.2350.5780.7620.0080.0660.1820.0020.0500.169

Mean pairwise errors: rotation 10.7° (val) / 8.3° (test), translation direction 42.1° / 40.2°.

GT vs predicted geometry on four validation scenes (outputs/vggt_train/visualization/index.html, view 0 shown on the right panel):

Scene 0: ground truth
Scene 0: predicted cameras + depth
Scene 1: ground truth
Scene 1: predicted cameras + depth
Scene 2: ground truth
Scene 2: predicted cameras + depth
Scene 3: ground truth
Scene 3: predicted cameras + depth

Errors on held-out scenes.

  1. Depth is the strongest part. δ1 is about 0.82 on both splits and the overall room layout (floor, walls, the big furniture) is right. This is the task closest to what DINOv2 features already capture. The predicted depth maps are smooth: thin structures such as the chair legs and backs in scene 0 disappear into the floor and wall, and edges have a blocky 14×14 patch pattern, because each token predicts one patch and the small refinement network only partly sharpens it. Some errors are also global scale errors, since depth is in normalized units and the scale has to be inferred from the images.
  2. Rotation is reasonable, translation is the weak part. Rotation AUC@30 is 0.74 / 0.76, but translation AUC@30 is only 0.18 and the mean translation direction error is about 40°. Within a 32-frame window of a handheld iPhone clip the camera mostly rotates and moves only a few centimetres, so the direction of the baseline is poorly constrained by the images and small absolute errors in the camera center become large angular errors. In scene 0, for example, the predicted camera centers are bunched together near the first camera, while the GT centers are more spread out, so the model under-estimates the baselines. Joint pose AUC is limited by translation (it takes the worse of the two errors per pair).
  3. Generalization gap is small. Validation and test numbers are very close and the validation loss was still slowly going down at the end, so the model is capacity- and data-limited (224 training scenes, frozen encoder) rather than overfitting. Compared with the overfit run (pose AUC@30 0.995), the gap shows how hard relative pose is on unseen rooms.

3.3 Bonus: number of input views

Question. The model is trained with 8 views. How do accuracy, runtime and memory change if it is given fewer or more views at test time?

Setup. The same best.pt from Q3.2, evaluated on the full validation split with V = 2, 4, 8 and 16 views from the first 32 prepared frames (224px, seed 825), using the provided evaluate. Runtime is the mean of 10 forward passes on one scene (batch size 1) after a warm-up, and memory is torch.cuda.max_memory_allocated during those passes, on an L40S (script: a4_tools/vggt_num_views.py). The V = 8 row matches the official Q3.2 evaluation.

ViewsPose pairsAbsRel ↓δ1 ↑Rot AUC@5Rot AUC@30Transl AUC@30Pose AUC@30Forward (ms)Peak memory (GB)
V = 2280.1160.8710.2640.7700.2380.23333.50.74
V = 41680.1300.8510.1920.7250.1910.17733.70.76
V = 87840.1450.8200.1890.7360.1780.17034.00.80
V = 1633600.1780.7820.1780.6920.1780.15938.00.88
number of views
Accuracy, runtime and memory against the number of input views.

Results.

  1. Accuracy goes down with more views. Depth AbsRel goes from 0.116 (V = 2) to 0.178 (V = 16), δ1 from 0.871 to 0.782, and pose AUC@30 from 0.233 to 0.159. I expected more views to help, as in the full VGGT. Two effects work against it here. First, with more views from the same 32 frames, more of the selected frames are far from frame 0, and both depth (in the reference frame and the shared scale) and relative pose are harder for distant frames. With V = 2 there is only 1 pair per scene; with V = 16 there are 120 pairs, many with wide baselines. So the per-V metrics are not over the same frames or pairs. Second, the model was only trained with 8 views, so V = 16 doubles the global-attention sequence length beyond anything seen in training, which is a distribution shift for the attention.
  2. Runtime and memory barely change at this scale. Forward time goes from 33.5 to 38.0 ms and peak memory from 0.74 to 0.88 GB. Most of the memory is the weights (DINOv2 ViT-B and the geometry transformer), and with 261 tokens per view even 16 views is only about 4200 tokens for global attention. With batch size 1 the GPU is underused and the time is dominated by kernel launches and the per-view DINO encoding, so the quadratic cost of global attention only starts to show between V = 8 and 16. For much larger V the global attention (O((V·N)2)) and activations would start to dominate.

Limitations. The frame subset changes with V, so this mixes the effect of the number of views with the effect of which frames are chosen. A cleaner version would fix a set of target frames and only add context views around them, or train with a random number of views. Timing is for inference with batch size 1 on one GPU and does not include training memory. There is a single checkpoint and seed.