16-825 Assignment 4: Multi-View Transformers
Data preprocessing · Mini-LVSM novel-view synthesis · Mini-VGGT cameras and depth. Everything here uses the ScanNet++ iPhone data at 224 px, and I ran all the experiments on RTX 3090s.
Part 1. Data Preprocessing
Unprojection

Canonicalization

check_data.py output
split: train, dataset_samples: 224, checked_samples: 4 sample views res valid_depth max_reproj_err(px) max_ray_reproj_err(px) centroid_dist e050c15a8d_frame_007080 8 224 0.978 1.13e-4 1.09e-4 1.0000 5f99900f09_frame_002820 8 224 0.990 1.23e-4 1.11e-4 1.0000 47b37eb6f9_frame_002720 8 224 0.961 0.99e-4 0.89e-4 1.0000 4c5c60fa76_frame_006690 8 224 0.752 1.16e-4 1.07e-4 1.0000 tensor shapes: images [2,8,3,224,224], ray_directions [2,8,3,224,224], plucker [2,8,6,224,224]
The full JSON is in outputs/check_data.txt. Every reprojection error is around 1e-4 px, which is basically float32 round-off and far under the 0.01 px threshold.
Explanations
- Canonicalization. The raw ScanNet++ poses live in whatever world frame the scan happened to start in, and the units are meters, so a big room and a small room look very different. Neither of those things can actually be inferred from the images. So I move everything into the first camera's frame (
T'i = T0-1 Ti, which makes camera 0 the identity), and then divide the translations, depths and points by the scene scale (distance from camera 0's center to the centroid of the valid points). After this every sample has the same convention and roughly unit scale, so the network isn't being asked to guess an absolute position or a metric size that the pixels just don't contain. - World-space ray directions. For a pixel (u, v),
K-1[u, v, 1]Tgives the direction it looks along in camera coordinates (OpenCV convention: x right, y down, z forward). Multiplying by the camera-to-world rotation R puts it in the shared world frame, and I normalize it:d = normalize(R K-1[u,v,1]T). The ray starts at the camera centert. One thing to be careful about is that the depth is z-depth, not distance along the ray, so for unprojection I use the unnormalized direction:X = t + depth · R K-1[u,v,1]T. - Plücker moment. Each ray is stored as
[d, m]wherem = o × d. The nice part is that m doesn't care which point on the line you pick as the origin, because(o + s d) × d = o × d(thed × dterm is zero). So the 6 numbers describe the line itself, not one particular point on it. Geometrically, since d is unit length,|m|is the distance from the world origin to the line, and m points along the normal of the plane through the line and the origin. Doing this for every pixel gives a smooth 6-channel map that encodes both the camera pose and the intrinsics, and that's what gets fed into the transformers.
Part 2. Novel-View Synthesis with Mini-LVSM
This is the decoder-only version of LVSM. For each source view I stack the RGB (shifted to [-1, 1]) with its Plücker rays, so 9 channels, and patchify it. The target view only gets its Plücker rays (6 channels), since that's the image we're trying to predict. Both sides get a 2-D sin-cos position embedding. Every target view becomes its own sequence [all source tokens, target tokens] and goes through 8 pre-norm transformer blocks with QK-norm. At the end I take just the target tokens, run them through a linear head and a sigmoid, and unpatchify them back into an image. The loss is plain pixel MSE.
Q2.1 Overfitting
The overfit run ended at a validation PSNR of 30.92 dB (SSIM 0.870), so the model can clearly memorize a scene.

outputs/lvsm_overfit/val_epoch000.png. Columns: source views, ground truth, prediction.Q2.2 Training and evaluation
Training settings. I kept the defaults from configs/lvsm.json:
- Model: patch 8, d_model 384, 8 layers, head_dim 64, MLP ratio 4, QK-norm.
- Data: 2 source views + targets from the default data config, batch 4.
- Optimizer: AdamW, lr 2e-4 with β = (0.9, 0.95), weight decay 0.05, 200 warmup steps then cosine decay, grad-clip 1.0, bf16 autocast.
I started with the default 100 epochs (0.7 h). When I looked at the curves they were still going up, and validation loss was below training loss, so the model was obviously underfitting. Because of that I trained the main model for 400 epochs instead (2.7 h on one RTX 3090), with everything else the same. I report both runs below. best.pt is the checkpoint with the lowest validation loss.

Validation metrics for the 400-epoch model, PSNR (dB) / SSIM:
| Source views | correct rays | shuffled rays | no rays (zeros) |
|---|---|---|---|
| 1 | 15.89 / 0.589 | 11.65 / 0.482 | 12.41 / 0.531 |
| 2 | 17.17 / 0.599 | 11.25 / 0.460 | 12.37 / 0.528 |
| 4 | 18.35 / 0.621 | 11.29 / 0.456 | 12.38 / 0.539 |
And the default 100-epoch model, just for comparison:
| Source views | correct | shuffled | none |
|---|---|---|---|
| 1 | 15.19 / 0.558 | 11.80 / 0.487 | 12.47 / 0.529 |
| 2 | 16.13 / 0.560 | 11.49 / 0.472 | 12.43 / 0.528 |
| 4 | 16.87 / 0.575 | 11.58 / 0.473 | 12.44 / 0.538 |
Training visualizations (columns: source views, ground truth, prediction):


Evaluation visualizations (400-epoch model, columns: source views, ground truth, prediction):









Discussion.
- Number of source views. Adding source views helps every time: 15.9 → 17.2 → 18.3 dB going from 1 → 2 → 4 views. What surprised me a bit is that this works even though the model only ever saw 2 views during training. With more views, more of what the target camera sees is actually visible in some source image, and the geometry is better pinned down, so the model can interpolate instead of making things up. Since attention just runs over however many source tokens there are, the model handles a different number of views without any retraining.
- Ray conditioning. If I zero out the target rays, PSNR drops by about 5–6 dB. Interestingly, that number is the same (≈12.4 dB) no matter how many source views there are. Without knowing where the target camera is, the best the model can do is output a blurry "average view" of the room. Shuffled rays are even worse than no rays (11.3 dB, and the lowest SSIM), because now the model confidently renders the scene from the wrong viewpoint. So the model really is using the Plücker rays to decide where to look, and isn't just copying one of the source images.
- Quality. The predictions get the rough layout, the colors and the big objects in the right place, but they're blurry and you can see the 8×8 patch artifacts. I think that's about what you'd expect from a small model (≈15M parameters) trained for a few GPU-hours on 224 training sequences with MSE, since MSE tends to average over anything uncertain. The curves hadn't converged yet, so more training, or a perceptual/SSIM loss, would probably make the outputs sharper.
Q2.3 Bonus: training view count
Question. If I change how many source views the model sees during training, how does that change its results when I evaluate with 1, 2 and 4 source views?
Setting. I trained three models that are identical except for num_source_views (1, 2 or 4): same default config, 100 epochs, same seed and schedule. Then I evaluated each one on the same validation set at S = 1, 2 and 4, using the correct rays.
| Train S \ Eval S | 1 | 2 | 4 | train time |
|---|---|---|---|---|
| 1 | 15.12 / 0.559 | 14.90 / 0.553 | 14.64 / 0.561 | 0.52 h |
| 2 | 15.19 / 0.558 | 16.13 / 0.560 | 16.87 / 0.575 | 0.68 h |
| 4 | 15.31 / 0.556 | 16.41 / 0.562 | 17.45 / 0.580 | 0.87 h |




Findings.
- The model trained with one view can't make use of extra views at all. Its PSNR actually goes down, from 15.1 to 14.6 dB, as I add views. It never had to combine information from several images, so the extra source tokens just get in the way.
- Once the model is trained with 2 or more views, it learns to fuse across views, and that carries over to more views than it was trained on.
- The 4-view model is the best at every evaluation setting, even S=1 (0.1–0.6 dB better than the 2-view model). My guess is that each training step gives it a stronger multi-view signal, and the target is usually well covered by the sources, so the gradients teach it something useful instead of pushing it to hallucinate content it can't see.
- The downside is longer sequences: for the same number of epochs, the S=4 model takes about 1.3× as long to train as S=2.
Limitations. These are only 100-epoch runs (all of them still underfitting) and one seed each, so the small gaps at S=1 (≈0.2 dB) are within run-to-run noise. I didn't try sampling a random number of views per step, but I'd expect that to get the robustness of S=1 together with the fusion ability of S=4.
Part 3. Cameras and Depth with Mini-VGGT
Here's a quick overview of the model:
- Encoder: I use frozen DINOv2 ViT-B/14 patch features, project them down to 384-d, and add a camera token and register tokens for each view.
- Aggregator: 8 pairs of alternating attention. Frame attention only looks inside one view, and global attention looks at all tokens from all views together. The outputs of the last frame block and the last global block get concatenated.
- Camera head: outputs a 9-d pose encoding per view (translation, quaternion and FoV).
- Depth head: a DPT-style dense head, refined with the RGB image, that outputs log-depth plus a confidence.
The losses are:
- Camera: L1 on the translation, quaternion and FoV.
- Depth: confidence-weighted regression (trimmed mean of
c·|Δ| − 0.2·log c) plus a gradient-matching term.
Q3.1 Overfitting a single scene
After 40 epochs on a single scene the camera loss is down to 0.0026. The total loss actually goes negative (−0.15), which is fine: it's just the −log(confidence) term once the model gets confident. The predicted geometry lines up with the GT, and the predicted centroid distance comes out at 0.999.



Q3.2 Training and evaluation
Training settings. Again I used the defaults, this time from configs/vggt.json:
- Data: 8 views sampled within a 32-frame range, 224 px, batch 16.
- Model: embed 384, 6 heads, 8 layers, frozen DINOv2 ViT-B/14.
- Optimizer: AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs, cosine decay to 0.05×.
- Loss weights: camera 5, depth 1.
- Schedule: 1000 epochs (14 steps each, 224 training sequences) in bf16 on one RTX 3090, which took 4.4 h. The best checkpoint by validation loss was from epoch 911.
For evaluation I use 8 views taken from the first 32 prepared frames, at 224 px with seed 825, over the full splits.

| Split | AbsRel ↓ | δ1 ↑ | Rotation AUC ↑ | Translation AUC ↑ | Joint pose AUC ↑ | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| @5 | @15 | @30 | @5 | @15 | @30 | @5 | @15 | @30 | |||
| val | 0.154 | 0.814 | 0.245 | 0.567 | 0.735 | 0.025 | 0.085 | 0.196 | 0.011 | 0.072 | 0.180 |
| test | 0.167 | 0.770 | 0.213 | 0.536 | 0.716 | 0.014 | 0.054 | 0.157 | 0.004 | 0.034 | 0.137 |
Each split has 28 samples and 784 pose pairs, and none of the pairs are degenerate. The mean rotation error is 11.3° on val and 10.1° on test; the mean translation-direction error is 42.2° and 42.9°.
GT vs. prediction on four validation scenes (left is the GT cameras + depth, right is what the model predicted):








Discussion of held-out errors.
- Depth works pretty well. δ1 is 0.81 / 0.77 and AbsRel is 0.15 / 0.17 (val / test). The predicted point clouds get the room layout right (floor, walls, furniture), and the depth maps are smooth with the right front-to-back ordering. Most of the error is at depth edges, which get smoothed out too much. In scene 0, for example, the table and chairs blur into one blob, and thin things like chair legs and chair backs disappear. I think a lot of the depth quality comes from the frozen DINOv2 features, which already know a lot about monocular depth.
- Rotation is decent. 73% AUC@30 on val, with an 11° mean error.
- Translation direction is where it struggles. AUC@30 is only around 0.2, and since joint pose AUC uses the worse of the two errors, translation is what holds it back. I think there are a few reasons:
- The views come from 32 consecutive iPhone frames, so the camera barely moves compared to how far away the scene is (you can see all the frusta bunched together in the screenshots). With such a short baseline, the translation direction is really sensitive: a tiny position error turns into a big angle error.
- The model has to figure out translation from parallax alone.
- There are only 224 training sequences. The val camera loss was still slowly going down at epoch 1000, so more data or more training should help.
- Generalization. Test is a little worse than val for both depth and pose, which makes sense given that the rooms are different. Training loss stays above validation loss, but that's because training samples random subsets of frames while evaluation uses a fixed and easier subset. It doesn't mean the model is underfitting.
Q3.3 Bonus: number of input views
Question. How does the number of input views change accuracy, speed and memory?
Setting. I took the model trained with 8 views and evaluated it on the val split with 2, 4, 8 and 16 views, all drawn from the first 32 prepared frames (same seed 825, 224 px). For speed and memory, I timed a single forward pass (batch 1, bf16, RTX 3090, averaged after a warm-up) with anywhere from 2 to 32 views.
| Views | AbsRel ↓ | δ1 ↑ | Rot AUC@15 | Rot AUC@30 | Trans AUC@30 | Pose AUC@30 | pairs |
|---|---|---|---|---|---|---|---|
| 2 | 0.117 | 0.857 | 0.600 | 0.746 | 0.229 | 0.187 | 28 |
| 4 | 0.137 | 0.851 | 0.548 | 0.717 | 0.186 | 0.157 | 168 |
| 8 (train) | 0.154 | 0.814 | 0.567 | 0.735 | 0.196 | 0.180 | 784 |
| 16 | 0.185 | 0.755 | 0.516 | 0.706 | 0.167 | 0.148 | 3360 |
| Views | 2 | 4 | 8 | 16 | 24 | 32 |
|---|---|---|---|---|---|---|
| ms / forward | 25.8 | 27.0 | 25.5 | 41.4 | 60.1 | 83.8 |
| ms / view | 12.9 | 6.8 | 3.2 | 2.6 | 2.5 | 2.6 |
| peak memory (GB) | 0.74 | 0.76 | 0.80 | 0.89 | 0.97 | 1.05 |

Findings.
- More views don't help this model, and depth actually gets worse. AbsRel goes up from 0.117 to 0.185 going from 2 to 16 views. Pose AUC stays roughly flat, maybe trending down slightly.
- Train/test mismatch. The model only ever saw 8 views in training. With 16 views, global attention is spread over twice as many tokens, which it's never seen before, so each view's depth gets mixed with more tokens that aren't very related to it.
- The settings aren't equally hard. Picking more views from the same 32-frame window gives more pairs, including wider-baseline pairs that are harder for pose. With only 2 views, the depth number is mostly just the reference view and the frame next to it.
- Depth doesn't really use the other views. It's mostly predicted from each image on its own using the DINO features, so extra views don't add much.
- Speed and memory are not a problem at this size. Up to 8 views the forward pass takes about 26 ms no matter what, because it's dominated by kernel launch overhead. After that, time grows roughly linearly, since the frozen DINOv2 encoder runs once per view and becomes the main cost. The global attention is quadratic in the 256·V tokens, but at V ≤ 32 that's still small. Time per view drops from 12.9 ms to 2.6 ms, and peak memory only goes from 0.74 to 1.05 GB, most of which is the DINOv2 weights.
Limitations. I only changed the number of views at evaluation time. Training with a random number of views (say 2–16) would probably fix the drop at 16 views, and that's the experiment I'd try next. Also, each view count is scored over a different set of pairs, so the numbers aren't strictly apples-to-apples.