16-825 Assignment 4: Multi-View Transformers
Farnaz Alam Ahmed
1. Multi-View Data Preprocessing (20 points)
Each training sample is built from 8 views of one ScanNet++ scene. Each view is a 400×300 RGB image with depth. It is center-cropped to 300×300 and resized to 224×224 (Q1.1). The depth is unprojected to 3D points (Q1.2). Everything is expressed in the first camera's frame and normalized in scale (Q1.3). Finally, a ray (Q1.4) and its Plücker coordinates (Q1.5) are computed for every pixel.
1.2 Depth unprojection (original world frame, meters)
Each pixel \((u, v)\) with z-depth \(Z\) is unprojected as \(\mathbf{X}_\text{world} = R\,(Z\,K^{-1}[u, v, 1]^\top) + \mathbf{t}\), using the resized intrinsics and the camera-to-world pose. Points from all 8 views fuse into a single consistent scene, and each camera frustum sits where its view was captured.
079a326597 (bedroom): points and cameras in the original world frame (m).
07ff1c45bb (office): points and cameras in the original world frame (m).1.3 Canonicalized geometry (first-camera frame, normalized units)
After canonicalization, camera 0 is the identity, so its center is at the origin. The valid-point centroid lies exactly 1 unit from it. In the screenshots below, the red diamond marks the first camera center and the yellow diamond marks the valid-point centroid. The black line between them is labeled with the centroid distance, and the highlighted box on the right repeats both values from the visualization page. The point cloud is slightly faded so that the markers stay visible.
079a326597 (bedroom): first camera center (0.000, 0.000, 0.000); valid-point centroid distance 1.000; scene scale 2.080 m.
07ff1c45bb (office): first camera center (0.000, 0.000, 0.000); valid-point centroid distance 1.000; scene scale 1.966 m.Canonicalization summary for the 5 visualized scenes
| Sample | First camera center (X, Y, Z) | Valid-point centroid distance | Scene scale (m) |
|---|---|---|---|
036bce3393_frame_004470 | (0.0000, 0.0000, 0.0000) | 1.000 | 1.867 |
076c822ecc_frame_002890 | (0.0000, 0.0000, 0.0000) | 1.000 | 2.830 |
079a326597_frame_007430 | (0.0000, 0.0000, 0.0000) | 1.000 | 2.080 |
07f5b601ee_frame_003630 | (0.0000, 0.0000, 0.0000) | 1.000 | 0.924 |
07ff1c45bb_frame_000460 | (0.0000, 0.0000, 0.0000) | 1.000 | 1.966 |
The scene scale is the centroid distance in meters before normalization. It is the divisor applied to translations, depth and points.
1.7 Final data check
| Sample | Valid depth | Max point reproj. error (px) | Max ray reproj. error (px) | Centroid distance |
|---|---|---|---|---|
e050c15a8d_frame_007080 | 97.8% | 1.10 × 10⁻⁴ | 9.39 × 10⁻⁵ | 1.0000 |
5f99900f09_frame_002820 | 99.0% | 1.11 × 10⁻⁴ | 1.11 × 10⁻⁴ | 1.0000 |
47b37eb6f9_frame_002720 | 96.1% | 9.59 × 10⁻⁵ | 8.90 × 10⁻⁵ | 1.0000 |
4c5c60fa76_frame_006690 | 75.2% | 1.09 × 10⁻⁴ | 1.01 × 10⁻⁴ | 1.0000 |
All checks pass.
Every reprojection error is about 10⁻⁴ px, roughly 100× below the 0.01 px requirement. What remains is float32 rounding.
The centroid distance is 1 for every sample. All batch shapes match the data contract
(batch 2, 8 views, 224×224): rgb, points_world and the rays are [2, 8, 3, 224, 224],
plucker is [2, 8, 6, 224, 224], K is [2, 8, 3, 3] and
T_world_from_camera is [2, 8, 4, 4].
Full check_data.py output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011009880108758807,
"max_ray_reprojection_error_px": 9.394258086103946e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.591897105565295e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010141761595150456,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Explanations
Canonicalization
Raw ScanNet++ poses live in an arbitrary per-scan world frame, and each room has its own physical size. A model would otherwise have to infer an absolute position and scale it cannot observe. Canonicalization removes both. First, every camera-to-world pose is re-expressed relative to the first sampled camera: \(T'_i = T_0^{-1}\,T_i\), computed in float64. This makes camera 0 the identity, at the origin and looking down \(+Z\). Depth is then unprojected in this frame, and the valid-point centroid distance \(s = \lVert \operatorname{mean}(\mathbf{p}_\text{valid}) \rVert\) is computed. Translations, depth and points are all divided by \(s\), and rotations are left unchanged. The centroid is not subtracted, so camera 0 stays at the origin and the scene's centroid sits exactly 1 unit away. Every training sample therefore shares the same reference frame and scale.
World-space ray directions
For pixel \((u, v)\), \(K^{-1}[u, v, 1]^\top\) gives the viewing direction in camera coordinates, with \(z = 1\). Rotating it by the camera-to-world rotation and normalizing gives the world-space unit direction \(\mathbf{d} = \dfrac{R\,K^{-1}[u, v, 1]^\top}{\lVert R\,K^{-1}[u, v, 1]^\top \rVert}\). Only the rotation is applied: a direction says which way the pixel looks, not where the camera is. The ray origin \(\mathbf{o}\) is the camera center \(\mathbf{t}\), shared by every pixel of that view. Every surface point seen by the pixel lies on \(\mathbf{o} + \lambda\mathbf{d}\), \(\lambda \ge 0\). This is what the ray reprojection check verifies.
Plücker moment
The Plücker coordinates of a ray are \([\mathbf{d}, \mathbf{m}]\) with moment \(\mathbf{m} = \mathbf{o} \times \mathbf{d}\). The moment does not depend on which point of the line is used as the origin: \((\mathbf{o} + \lambda\mathbf{d}) \times \mathbf{d} = \mathbf{o} \times \mathbf{d}\), because \(\mathbf{d} \times \mathbf{d} = 0\). So the 6-vector describes the line itself, not one point on it. Geometrically, \(\mathbf{m}\) is normal to the plane containing the ray and the origin, and with unit \(\mathbf{d}\), \(\lVert \mathbf{m} \rVert\) is the ray's perpendicular distance from the origin. Camera 0 sits at the canonical origin, so all of its rays have \(\mathbf{m} = 0\). Other cameras' moments encode where they are relative to camera 0. This dense 6-channel map is the camera conditioning used by the models in Parts 2 and 3.
2. Novel-View Synthesis with Mini-LVSM (40 points)
2.1 Overfitting one batch
The model was trained for 2000 steps on a single fixed training batch, with the same model and optimizer settings as 2.2. It reached a final PSNR of 30.99 dB (SSIM 0.872, MSE 8.5 × 10⁻⁴) on that batch, which passes the ~30 dB sanity check. The averaged training PSNR over all 2000 steps was 25.40 dB (it starts near 12 dB). The run took 717 s on an NVIDIA A10G.
val_epoch000.png from the overfit run. Columns: the 2 source views, ground-truth target and prediction.
The prediction reproduces the memorized target's layout and colors, with only slight blur at fine detail.2.2 Training and evaluation
Training settings
| Model | 14.6M parameters; patch size 8 (784 tokens per view); width 384; 8 layers; head dim 64; MLP ratio 4; QK-norm |
|---|---|
| Data | ScanNet++ iPhone, 224×224, 8 views per sample: 2 source views and 6 target views; 224 training clips |
| Optimization | AdamW, lr 2 × 10⁻⁴, betas (0.9, 0.95), weight decay 0.05; 200 warmup steps, then cosine decay to 10%; gradient clip 1.0 |
| Schedule | Batch size 4, 100 epochs (56 steps per epoch, 5600 steps); bf16 autocast; seed 825 |
| Hardware | One NVIDIA A10G (24 GB, AWS g5.xlarge); ~28 s per epoch, 46 min in total |
Training curves
Validation PSNR rises from 12.2 dB to 16.13 dB and SSIM from 0.38 to 0.561; the best checkpoint is the final epoch. Train and validation curves stay close (final training PSNR 15.60 dB), so the model is not overfitting. Both curves are still rising slowly at epoch 100, so longer training would likely help further.
Training visualizations
After epoch 0 the predictions are a nearly uniform gray. By epoch 95 they recover the coarse color layout of each held-out target: bright windows and walls, dark furniture and floor in roughly the right places. Object shapes, fine texture and thin structures are still very blurry and patch-blocky.
Ablations: number of source views × target-ray conditioning
The best checkpoint is evaluated on the validation split with 1, 2 or 4 source views, using the remaining views as targets. Target rays are either correct, shuffled (the rays of a different target view) or none (all zeros).
| Source views | Correct rays | Shuffled rays | No rays (zeros) | Target images | |||
|---|---|---|---|---|---|---|---|
| PSNR (dB) | SSIM | PSNR (dB) | SSIM | PSNR (dB) | SSIM | ||
| 1 | 15.19 | 0.558 | 11.80 | 0.488 | 12.48 | 0.527 | 196 |
| 2 | 16.13 | 0.561 | 11.49 | 0.472 | 12.46 | 0.526 | 168 |
| 4 | 16.89 | 0.576 | 11.59 | 0.473 | 12.47 | 0.536 | 112 |
Evaluation visualizations
Columns: source views, ground truth, prediction. Each image shows the first validation clips.





Remaining ablation images




Discussion
Number of source views. With correct rays, quality improves steadily with more source views: 15.19 → 16.13 → 16.89 dB, although the model was trained only with 2. More views cover more of the scene, so less of the target has to be guessed. The model can use them because attention takes any number of source tokens, and the Plücker rays tell it where each source pixel sits relative to the target camera. The gain shrinks per added view (+0.94 dB, then +0.76 dB for two more), as views increasingly overlap.
Ray conditioning. The target rays are the only information about which view to render. With all-zero rays, PSNR falls to about 12.5 dB for every source count, and adding views no longer helps. The model then outputs a blurry average-looking image, which minimizes MSE when the viewpoint is unknown. Shuffled rays are worse still (about 11.5 dB, lowest SSIM). The model trusts the rays and renders a confident but wrong viewpoint, and a sharp wrong image costs more MSE than a blurry average one. Together these show that the model really does use the camera geometry, rather than memorizing scene appearance.
Limitations. Predictions are blurry and show 8×8 patch seams. Small model, 100 epochs on 224 clips, and the MSE loss all favor smooth averages of plausible outputs. Regions not visible in any source view cannot be recovered and come out as a smooth guess.
2.3 Bonus: training with a random number of source views
Question. The baseline always trains with S = 2 source views, yet it is evaluated with 1, 2 and 4. Does training with a random S ∈ {1, 2, 3, 4} at each step (remaining views as targets) make the model better across view counts, especially for S = 1 and S = 4, which the baseline never saw?
Setup. A controlled comparison: identical model, optimizer, schedule, 100 epochs, data and seed (825).
The only change is that each training step draws S uniformly from {1, 2, 3, 4}
(config configs/lvsm_randS.json, key train_num_source_views). Validation during training still uses S = 2,
so the best checkpoint is chosen the same way. Both checkpoints are evaluated with the same 9-setting ablation.
| Evaluation setting | Baseline (S = 2) PSNR / SSIM | Random S PSNR / SSIM | Δ PSNR (dB) |
|---|---|---|---|
| S = 1, correct | 15.19 / 0.558 | 15.21 / 0.559 | +0.01 |
| S = 2, correct | 16.13 / 0.561 | 16.19 / 0.559 | +0.06 |
| S = 4, correct | 16.89 / 0.576 | 17.00 / 0.573 | +0.12 |
| S = 1, shuffled | 11.80 / 0.488 | 11.79 / 0.487 | −0.01 |
| S = 2, shuffled | 11.49 / 0.472 | 11.46 / 0.469 | −0.03 |
| S = 4, shuffled | 11.59 / 0.473 | 11.51 / 0.466 | −0.08 |
| S = 1, none | 12.48 / 0.527 | 12.46 / 0.532 | −0.02 |
| S = 2, none | 12.46 / 0.526 | 12.43 / 0.528 | −0.03 |
| S = 4, none | 12.47 / 0.536 | 12.46 / 0.537 | −0.01 |


Results. The two models are nearly identical. The validation curves overlap throughout training. The largest difference is +0.12 dB at S = 4, the setting the baseline never trained on, and S = 1 is unchanged. SSIM differs by under 0.01 everywhere, and the shuffled/no-ray settings are unchanged; the images are visually indistinguishable.
Interpretation. Training with varying S gives at most a small benefit for more views than trained with, and nothing for fewer. The baseline already generalizes across view counts. Attention is agnostic to sequence length, and every source token carries its own Plücker rays, so the per-token computation does not depend on how many other sources are present. The small model is capacity-limited (~16 dB), and that bottleneck dominates any effect of the view-count curriculum.
Limitations. One seed per configuration. Differences of ~0.1 dB are likely within run-to-run noise, so the S = 4 gain is only suggestive. A firmer answer needs several seeds, longer training, a larger model, and harder settings (for example, S up to 7 or wider baselines), where a fixed-S model might generalize less well.
3. Multi-View Geometry with Mini-VGGT (40 points)
3.1 Overfitting one scene
One fixed training clip (036bce3393_frame_004470, 8 views) was fitted for 40 epochs × 25 steps, with learning rate 10⁻³,
on an NVIDIA A10G (~3 s per epoch).
All three losses fall steadily. The total loss ends below zero, which is expected: once predictions are accurate, the confidence term's −0.2·log(conf) reward outweighs the remaining weighted error.
| Metric on the fitted clip | Epoch 0 | Epoch 40 |
|---|---|---|
| Total loss | 5.48 | −0.16 |
| Camera loss / depth loss | 0.342 / 3.77 | 0.0027 / −0.175 |
| Mean rotation error | 49.7° | 0.20° |
| Mean translation-direction error | 90.0° | 0.43° |
| Pose AUC @ 5° / 15° / 30° | 0 / 0 / 0 | 0.986 / 0.995 / 0.998 |
| Depth AbsRel / δ1 | 0.619 / 0.208 | 0.036 / 0.956 |
The predicted geometry matches the ground truth closely: the camera trajectory, the room layout and the depth map all line up. The predicted first camera center is (0.0000, −0.0001, −0.0001), so the model has learned that view 0 is the identity. The main remaining difference is in the depth map, which shows faint 14×14 patch blocks from the per-patch decoder. It also fills in pixels that are missing (black) in the sensor depth, and those pixels are not supervised.
3.2 Training and evaluation
Training settings
| Model | Frozen DINOv2 ViT-B/14; width 384, 6 heads, 8 frame/global pairs (16 blocks); camera head with 4 blocks; depth head with RGB refinement |
|---|---|
| Data | ScanNet++ iPhone, 8 views sampled from a 32-frame range, 224×224; 224 training clips, 28 validation clips |
| Optimization | AdamW, lr 6 × 10⁻⁴, weight decay 0.01; 10 warmup epochs, then cosine decay to 5%; loss weights camera 5, depth 1 |
| Schedule | Batch size 16 clips (14 steps per epoch), 1000 epochs (default config); best checkpoint chosen by validation loss |
| Hardware | One NVIDIA A10G (24 GB, AWS g5.xlarge); ~15 GB GPU memory, 20–35 s per epoch including validation, ~9 h in total |
Training curves
Depth loss drops quickly in the first ~50 epochs and then flattens. Camera loss keeps decreasing slowly for the whole run. Validation loss is lower than training loss throughout. This is most likely because training clips use random view sampling within the 32-frame range, while validation uses a fixed, easier sampling of the first 32 prepared frames, not because of overfitting.
Validation and test metrics
Evaluated with best.pt: 8 views from the first 32 prepared frames, 224 px, seed 825, full splits (28 clips each, 784 view pairs).
| Split | Depth | Rotation AUC | Translation AUC | Joint pose AUC | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| AbsRel ↓ | δ1 ↑ | @5° | @15° | @30° | @5° | @15° | @30° | @5° | @15° | @30° | |
| Validation | 0.139 | 0.834 | 0.199 | 0.518 | 0.709 | 0.013 | 0.077 | 0.196 | 0.005 | 0.061 | 0.180 |
| Test | 0.151 | 0.797 | 0.219 | 0.522 | 0.710 | 0.009 | 0.072 | 0.183 | 0.002 | 0.048 | 0.163 |
Mean pairwise errors: rotation 11.8° (val) / 10.3° (test); translation direction 41.8° (val) / 40.5° (test). For comparison, the model at epoch 0 had depth AbsRel 0.73, and the overfit run reached 0.04 on its single clip.
Ground truth vs prediction on four held-out validation scenes
Each pair shows the GT (left) and the predicted cameras + depth (right) in the canonical first-camera frame. The viewpoint is behind the cameras, looking into the room, so the reconstructed surfaces face the viewer.
09c1414f1b: dining room


21d970d8de: office floor and chair


280b83fcf3: hallway




2b1dc6d6a5: office desk




Discussion of errors on held-out scenes
What works. On unseen rooms the model recovers the overall scene layout and scale. Floors, walls, the hallway and the desk area are in the right places, and the valid-point centroid distance of the predictions is 0.97–0.99 (GT: 1.00), so the predicted metric scale is nearly correct. Depth is good at the coarse level (δ1 ≈ 0.80–0.83, AbsRel ≈ 0.14–0.15), and relative rotations are mostly within 15–30° (rotation AUC@30 ≈ 0.71). Val and test are close, so the model generalizes consistently to new scenes.
Depth errors. Predicted depth is over-smoothed. Thin structures disappear (the chair legs and table edges
in 09c1414f1b), depth discontinuities are blurred into ramps, and flat walls become slightly wavy and thick in 3D.
The depth head decodes each 14×14 DINO patch with a linear layer plus a tiny convolutional refinement, so it cannot represent detail
finer than a patch. The L1-style loss also favors smooth averages where it is uncertain.
Camera errors. Translation direction is the weakest output (AUC@30 ≈ 0.19, mean error ~41°), and it drags down the joint pose AUC. Views come from at most 32 consecutive video frames, so baselines are short and often dominated by rotation. For a short baseline, a small absolute error in translation becomes a large error in its direction. The model also has to infer translation purely from parallax in frozen DINO features, without iterative refinement. Rotation is easier: it follows directly from how image content shifts between views. Fine accuracy is low everywhere (AUC@5 ≤ 0.22), so the model gets cameras roughly right but not precisely.
Limitations. A small model (width 384) trained only on 224 clips for ~9 h on one A10G. The camera loss was still decreasing at the end of training, so longer training would likely improve poses further. Larger gains would probably come from a stronger depth decoder (for example, DPT-style multi-scale features), iterative camera refinement, more training data, or unfreezing DINO.
3.3 Bonus: how does the number of input views affect accuracy, runtime and memory?
Question. Mini-VGGT was trained only with 8 views, but its architecture accepts any number of views: frame attention is per view, and global attention simply sees a longer sequence. Does the trained model get better with more views (more parallax and more context), worse with fewer, or does it break outside its training setting? And how do runtime and memory scale?
Setup. Evaluation only, with the 3.2 checkpoint (best.pt, epoch 890) and no retraining.
Full val and test splits, using the same protocol as 3.2 (first 32 prepared frames, 224 px, seed 825), with V ∈ {2, 4, 8, 12, 16, 24, 32}.
The evaluation loader always takes frame 0 plus the first V−1 frames of a fixed per-scene random order.
The view sets are therefore nested: every larger set contains the smaller ones, so V is the only thing that changes.
As a check, V = 8 reproduces the 3.2 numbers exactly.
Runtime and peak GPU memory were measured separately at batch size 1 on the A10G (bf16 autocast, 10 timed forward passes after warm-up;
training step = forward + backward with frozen DINO).
Accuracy vs number of views
| Views | Pose pairs | Validation | Test | ||||||
|---|---|---|---|---|---|---|---|---|---|
| AbsRel ↓ | δ1 ↑ | Rot AUC@30 | Trans AUC@30 | AbsRel ↓ | δ1 ↑ | Rot AUC@30 | Trans AUC@30 | ||
| 2 | 28 | 0.109 | 0.880 | 0.719 | 0.281 | 0.114 | 0.886 | 0.699 | 0.210 |
| 4 | 168 | 0.129 | 0.851 | 0.670 | 0.227 | 0.139 | 0.816 | 0.727 | 0.212 |
| 8 (trained) | 784 | 0.139 | 0.834 | 0.709 | 0.196 | 0.151 | 0.797 | 0.709 | 0.183 |
| 12 | 1848 | 0.151 | 0.806 | 0.691 | 0.172 | 0.154 | 0.795 | 0.702 | 0.193 |
| 16 | 3360 | 0.163 | 0.805 | 0.672 | 0.172 | 0.159 | 0.787 | 0.714 | 0.191 |
| 24 | 7728 | 0.155 | 0.800 | 0.677 | 0.169 | 0.155 | 0.799 | 0.721 | 0.191 |
| 32 | 13888 | 0.164 | 0.786 | 0.663 | 0.165 | 0.157 | 0.801 | 0.724 | 0.189 |
Runtime and memory vs number of views
| Views | Tokens in global attention | Inference (ms) | Inference peak memory (GB) | Training step (ms) | Training peak memory (GB) |
|---|---|---|---|---|---|
| 2 | 522 | 33 | 0.68 | 78 | 0.90 |
| 4 | 1,044 | 35 | 0.70 | 79 | 1.09 |
| 8 | 2,088 | 34 | 0.74 | 78 | 1.48 |
| 16 | 4,176 | 52 | 0.82 | 109 | 2.28 |
| 24 | 6,264 | 76 | 0.89 | 162 | 3.09 |
| 32 | 8,352 | 106 | 0.97 | 227 | 3.92 |
Results and conclusions
More views do not help. Depth is best with the fewest views: val AbsRel goes 0.109 → 0.139 → 0.164 for 2 → 8 → 32 views, with the same trend on test. Rotation AUC@30 stays at 0.66–0.73 with no clear trend, and translation AUC@30 falls on val (0.28 → 0.17) and is flat on test. The model does not use the extra parallax. It does degrade gracefully, though: even at 32 views (4× its training setting) no metric collapses, because attention with per-view 2D positions works for any number of views.
Possible reasons (not verified): extra views dilute global attention in a way the model never saw in training; with few views, the easier reference view 0 makes up a larger share of the depth metric; and pose AUCs at small V use few pairs, so they are noisy.
Cost. Inference time is flat (~34 ms) up to 8 views, then grows faster than linearly: 52 ms at 16 views, 106 ms at 32. Inference memory barely changes (0.7 → 1.0 GB). Training memory grows linearly, about 0.1 GB per view, so 32 views at batch size 16 would need about 60 GB, more than the 24 GB GPU.
Limitations. One checkpoint, trained at 8 views and evaluated only. A model trained with more or varying views might benefit from them. All views come from the same 32-frame window, so extra views add denser coverage, not wider baselines.