Assignment 4: Multi-View Transformers
Author: Po-Ting Ko
Andrew ID: potingk
1. Multi-View Data Preprocessing
1.1 Resizing camera intrinsics
The +0.5 is there because pixel centers sit at integer coordinates, so the image edge is at −0.5.
1.2 Depth unprojection
Points in the original world frame (meters). The red frustum is the first sampled camera.
1.3 Camera and depth canonicalization
Why canonicalize? A scan's world frame (origin, orientation, size) is arbitrary and cannot be predicted from images. Working relative to view 0 removes the rigid-transform ambiguity, and fixing the centroid distance to 1 removes the scale ambiguity.
The viewer's side panel, enlarged under each screenshot, shows the first camera center and the valid-point centroid distance. All 5 scenes give (0, 0, 0) and 1.000, with scene scales from 0.924 to 2.830 m.
1.4 Camera rays
World-space directions. Camera-space directions depend only on K, so they are the same for every view. In the shared frame, rays of different views become comparable: rays of pixels that see the same 3D point intersect there.
1.5 Plücker coordinates
Plücker moment. Every point o + λd on the line gives the same moment. It is perpendicular to the plane through the line and the origin, and its norm is the line's distance from the origin. So (d, m) identifies the line, and each pixel encodes both camera orientation and position.
1.7 Final check
| Sample | Valid depth | Point reproj. error (px) | Ray reproj. error (px) | Centroid distance |
|---|---|---|---|---|
e050c15a8d_frame_007080 |
0.978 | 1.11e-4 | 1.08e-4 | 1.000 |
5f99900f09_frame_002820 |
0.990 | 1.23e-4 | 1.11e-4 | 1.000 |
47b37eb6f9_frame_002720 |
0.961 | 9.78e-5 | 8.90e-5 | 1.000 |
4c5c60fa76_frame_006690 |
0.752 | 1.12e-4 | 1.02e-4 | 1.000 |
All reprojection errors are far below the 0.01 px limit, and the shapes match the data contract.
Full check_data.py output
$ python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011141905270051211,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.777836385183036e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011162471491843462,
"max_ray_reprojection_error_px": 0.00010225349979009479,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM
In every visualization, the columns are: source views, ground truth, prediction.
2.1 Overfitting
After 2,000 steps on one fixed batch, the final PSNR is 34.02 dB (SSIM 0.915).
2.2 Training and evaluation
Training settings: configs/lvsm_local_300ep.json. Same as configs/lvsm.json except batch 2 (batch 4 exceeds 8 GB) and 300 epochs (33,600 steps), because the default 100 epochs reached 16.16 dB and was still improving. Patch size 8, width 384, 8 layers, head dim 64, QK-norm (14.6M parameters). 2 source and 6 target views out of 8, sampling range 32. AdamW with lr 2e-4, betas (0.9, 0.95), weight decay 0.05, 200 warmup steps then cosine decay, gradient clip 1.0, bf16. Training took 3 h 40 min on an RTX 5060 Laptop GPU (8 GB). best.pt is from epoch 296.
| Source views | Correct rays: PSNR / SSIM | Shuffled rays: PSNR / SSIM | No rays: PSNR / SSIM |
|---|---|---|---|
| 1 | 15.69 / 0.580 | 11.67 / 0.483 | 12.49 / 0.538 |
| 2 | 16.86 / 0.586 | 11.26 / 0.462 | 12.45 / 0.534 |
| 4 | 17.89 / 0.604 | 11.30 / 0.461 | 12.48 / 0.544 |
Validation split, best.pt. As a baseline, copying the most similar source view gives only 11.9 dB (S = 2).
- Number of source views. More sources help, even though training used 2: 15.69 → 16.86 → 17.89 dB for S = 1, 2, 4. Each extra view covers more of the target.
- Ray conditioning. Shuffled rays cost 5.6 dB at S = 2 and are worse than none: the model renders a plausible but misplaced view, while no rays gives an almost uniform average color. Without correct rays, extra sources no longer help.
- Limitations. Predictions get the coarse layout right and follow the target camera, but are blurry with a visible 8×8 patch grid. The same model memorizes one batch at 34 dB, so the limit is capacity and training budget.
2.3 Bonus: adding an SSIM loss
Question. MSE averages over plausible images, which causes blur. Does an SSIM term give sharper novel views, and at what cost in PSNR?
Setup. The loss is MSE + λ (1 − SSIM), with λ = 0.05 and 0.2. On the trained model the SSIM gradient is about 7× the MSE gradient, so λ ≈ 0.14 balances them. Everything else matches a 100-epoch MSE control run. scripts/lvsm_sharpness.py measures two ratios on 168 validation targets: sharpness (mean image gradient of the prediction over that of GT) and blockiness (mean pixel difference across 8-px patch borders over that inside patches; GT gives 1.01).
| Loss (100 epochs) | S = 1: PSNR / SSIM | S = 2: PSNR / SSIM | S = 4: PSNR / SSIM | Sharpness ↑ | Blockiness ↓ |
|---|---|---|---|---|---|
| MSE (control) | 15.20 / 0.559 | 16.16 / 0.561 | 16.86 / 0.576 | 0.299 | 7.79 |
| MSE + 0.05 (1 − SSIM) | 14.49 / 0.567 | 15.01 / 0.569 | 15.23 / 0.581 | 0.153 | 4.07 |
| MSE + 0.2 (1 − SSIM) | 14.48 / 0.571 | 14.97 / 0.573 | 15.13 / 0.584 | 0.131 | 3.20 |
| MSE, 300 epochs (Q2.2) | 15.69 / 0.580 | 16.86 / 0.586 | 17.89 / 0.604 | 0.328 | 5.47 |
- Result. SSIM removes the patch grid (blockiness 7.8 → 3.2) but does not sharpen: sharpness halves, and PSNR drops 0.7–1.7 dB for an SSIM gain of about 0.01.
- Why. SSIM windows straddle patch borders, so jumps between patches are penalized, while MSE is per pixel. The model answers by blurring across borders. Both λ behave the same, so the SSIM term likely dominates early training, where the gradient balance differs from my estimate.
- Longer training helps more. The 300-epoch MSE model beats both SSIM runs on every metric except blockiness, where it sits between them and the control.
- Limitations. One seed and 100 epochs per run. A perceptual loss, or adding SSIM after MSE converges, may work better.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
One fixed clip, 40 epochs × 25 steps. At the end: rotation error 0.22° (from 49.7°), translation-direction error 0.35° (from 90°), depth AbsRel 0.037 (from 0.619). Total and depth losses go negative because of the −0.2 log c confidence term.
3.2 Training and evaluation
Training settings: configs/vggt_local.json. Same as configs/vggt.json except batch 4 × 4 gradient-accumulation steps, so each update still sees 16 samples (batch 16 does not fit in 8 GB; I added training.grad_accum_steps to models/vggt/train.py). Frozen DINOv2 ViT-B/14, 8 frame/global pairs, width 384, 6 heads (57.7M trainable parameters). 8 views within 32 frames, 224 px. 1,000 epochs with AdamW, lr 6e-4, weight decay 0.01, 10 warmup epochs then cosine decay to 5%, gradient clip 1.0, bf16. Training took 5.9 h on an RTX 5060 Laptop GPU (8 GB). best.pt is from epoch 797 (val loss 0.864).
Train loss is above validation loss because training samples 32-frame windows anywhere in a sequence, while validation always uses the first 32 frames.
| Split | AbsRel ↓ | δ1 ↑ | Rotation AUC @5 / 15 / 30 | Translation AUC @5 / 15 / 30 | Joint pose AUC @5 / 15 / 30 |
|---|---|---|---|---|---|
| Val | 0.140 | 0.846 | 0.177 / 0.479 / 0.679 | 0.018 / 0.076 / 0.175 | 0.005 / 0.055 / 0.154 |
| Test | 0.173 | 0.780 | 0.173 / 0.489 / 0.692 | 0.006 / 0.050 / 0.142 | 0.002 / 0.035 / 0.126 |
best.pt, 8 views from the first 32 frames, seed 825. 28 scenes and 784 view pairs per split.
Comparison with the TA reference. I ran my model on the same 8-view samples as the 4 scenes in the TA viewer (GT points match exactly) and computed the same metrics for both from the viewer data (scripts/compare_ta_reference.py):
| 4 TA scenes, mean | Rotation error ↓ | Translation-direction error ↓ | AbsRel ↓ | δ1 ↑ | Point L2 ↓ |
|---|---|---|---|---|---|
| TA reference | 33.7° | 46.8° | 0.182 | 0.778 | 0.530 |
| Mine | 28.4° | 40.5° | 0.148 | 0.841 | 0.449 |
The gain comes mainly from be0ed6b33c and fd361ab85f. On 394a542a19 and 7b6477cb95 the two models are close, and each wins some metrics. With 4 scenes this is a sanity check, not a ranking.
- Depth generalizes best (δ1 0.85 / 0.78). Floors and walls are correct, but thin structures become blobs and edges blur over about one 14-px patch. Much of the error is global scale: per-sample median scaling drops val AbsRel from 0.140 to 0.089.
- Rotation is moderate, with a mean error of 10.6–14.3°.
- Translation direction is the main error, at about 44°, and it limits joint pose AUC. Baselines within 32 frames are short, so small position errors flip the direction, and the L1 camera loss does not target direction. Misplaced cameras make the fused point clouds thick and duplicated.
- Limitations. The same model fits one clip to 0.2° and AbsRel 0.037, so generalization is the limit. Both my model and the TA's fail on
7b6477cb95(rotation error above 80°). More scenes or iterative camera refinement would be the next steps.
3.3 Bonus: number of input views
Question. The model is trained with 8 views. How do accuracy, runtime and memory change with V at test time?
Setup. The same best.pt with V ∈ {2, 4, 8, 16, 24, 32} from the first 32 frames, seed 825, no retraining. View sets are nested, so core metrics score only the 4 views every run shares, and only the extra context changes. Aligned AbsRel first scales each sample's depth by median(GT / prediction), separating depth shape from global scale. Runtime and memory use fixed inputs (batch 1, median of 20 runs) for V up to 128. One more sweep draws 8 views from the first 128 frames.
| V | All: AbsRel / aligned / pose AUC@30 | Core: AbsRel / aligned / pose AUC@30 | Time (ms) | Memory (GiB) |
|---|---|---|---|---|
| 2 | 0.110 / 0.081 / 0.163 | – | 27.0 | 0.70 |
| 4 | 0.131 / 0.086 / 0.132 | 0.131 / 0.086 / 0.132 | 30.8 | 0.71 |
| 8 (training) | 0.140 / 0.089 / 0.154 | 0.139 / 0.085 / 0.147 | 45.8 | 0.75 |
| 16 | 0.170 / 0.094 / 0.129 | 0.171 / 0.086 / 0.142 | 83.2 | 0.84 |
| 32 | 0.179 / 0.095 / 0.124 | 0.181 / 0.086 / 0.142 | 172.4 | 0.98 |
| 64 | – | – | 405.1 | 1.29 |
| 128 | – | – | 1054.2 | 1.89 |
Validation split. Test shows the same trends, with core aligned AbsRel 0.095–0.096 for every V.
- Depth. Extra views do not change depth shape but hurt scale: on the core views, aligned AbsRel stays at 0.086 while raw AbsRel rises from 0.131 to 0.181. The GT scale depends on the centroid of all V views, and the model, trained only with 8, misjudges it.
- Pose is roughly flat, with core AUC@30 at 0.13–0.15, within the noise of 168 pairs.
- Wider baselines break the model: 8 views over 128 frames give AbsRel 0.397 (aligned 0.114) and pose AUC@30 0.059, because training only covered 32-frame windows.
- Coverage. The point cloud at V = 32 covers much more of the room.
- Runtime and memory. Time is about linear from V = 8 to 32, then grows faster (2.6× from 64 to 128) as global attention over
V·Ntokens takes over. Memory grows only about linearly, because fused attention never stores the attention matrix. - Limitations. One checkpoint trained with V = 8 only, and 28 scenes per split. Training with random V might remove the scale drift.
































