16-825 · Learning for 3D Vision
036bce3393, original world frame (meters). Red frustum: first camera.
07f5b601ee, original world frame (meters).
036bce3393, canonical frame
07f5b601ee, canonical frame| Scene | First camera center | Centroid distance |
|---|---|---|
036bce3393 | (0, 0, 0) | 1.000 |
07f5b601ee | (0, 0, 0) | 1.000 |
| Sample | Max point reproj. (px) | Max ray reproj. (px) | Centroid distance |
|---|---|---|---|
e050c15a8d_frame_007080 | 1.13×10−4 | 1.09×10−4 | 1.000 |
5f99900f09_frame_002820 | 1.23×10−4 | 1.11×10−4 | 1.000 |
47b37eb6f9_frame_002720 | 0.99×10−4 | 0.89×10−4 | 1.000 |
4c5c60fa76_frame_006690 | 1.09×10−4 | 1.03×10−4 | 1.000 |
check_data.py output{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001089862416847609,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Canonicalization. Poses are expressed in the first camera's frame, \(T'_i = T_0^{-1} T_i\), and translations, depth and points are divided by the distance of the valid-point centroid. Camera 0 becomes the identity and the centroid lies at distance 1, so every sample shares one frame and scale.
World-space ray directions. \(\mathbf{d} = R\,K^{-1}[u, v, 1]^\top\), normalized to unit length, where \(R\) is the camera-to-world rotation. The ray origin is the camera center \(\mathbf{o} = \mathbf{t}\).
Plücker moment. \(\mathbf{m} = \mathbf{o} \times \mathbf{d}\) is the same for every point on the ray, so \([\mathbf{d}, \mathbf{m}]\) describes the ray itself. \(\lVert\mathbf{m}\rVert\) is the ray's distance from the origin, which is zero for camera 0.
Final overfit PSNR: 30.88 dB
val_epoch000.png. Columns: source 1, source 2, ground truth, prediction.| Setting | Value |
|---|---|
| Config | Default configs/lvsm.json, unchanged |
| Views | 8 per sample: 2 sources, 6 targets |
| Training | 100 epochs, batch size 4 (5,600 steps) |
| Optimizer | AdamW, lr 2×10−4, weight decay 0.05; 200-step warmup, cosine decay |
| Model | Patch 8, width 384, 8 layers; 14.6M parameters |
| Hardware | One RTX 4090, 30.5 min |
| Source views | Correct rays | Shuffled rays | No rays |
|---|---|---|---|
| 1 | 15.20 / 0.558 | 11.81 / 0.488 | 12.47 / 0.528 |
| 2 | 16.14 / 0.560 | 11.49 / 0.472 | 12.44 / 0.527 |
| 4 | 16.88 / 0.575 | 11.59 / 0.473 | 12.46 / 0.538 |
val_epoch095.png. Columns: source 1, source 2, ground truth, prediction.
Number of source views. With correct rays PSNR rises from 15.20 to 16.14 to 16.88 dB for 1, 2 and 4 sources. More sources cover more of the target view, so less has to be guessed. The model was trained with 2 sources but also works with 1 and 4, because attention accepts any number of source tokens.
Ray-conditioning ablation. Correct rays beat shuffled rays by 3.4–5.3 dB and zero rays by 2.7–4.4 dB, and only correct rays improve with more sources, so the model uses the target camera. Shuffled rays are worst: the model renders another view's layout in the wrong place, while with zero rays it falls back to a blurry scene average.
Question. The 2.2 predictions are blurry and show 8×8 blocks. Do a decoder that sees across patches and a perceptual loss (LPIPS) fix this, and how do PSNR, SSIM and sharpness change?
bonus/lvsm_plus.py): the linear head outputs 3 + 16 channels per pixel,
followed by four 3×3 conv layers that cross patch borders. Width 768, 12 layers (86.8M parameters),
batch size 8, lr 3×10−4, 400 epochs. The MSE+LPIPS run was then resumed to 1000 epochs
(learning-rate schedule restarted at epoch 400).| Model (val, 2 sources) | Epochs | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Sharpness | Blockiness |
|---|---|---|---|---|---|---|
| Baseline (2.2) | 100 | 16.14 | 0.560 | 0.672 | 0.33 | 7.97 |
| Baseline long | 1000 | 17.62 | 0.612 | 0.566 | 0.42 | 3.43 |
| LVSM+ MSE | 400 | 17.00 | 0.609 | 0.693 | 0.28 | 1.06 |
| LVSM+ MSE+LPIPS | 400 | 17.19 | 0.589 | 0.508 | 0.51 | 1.37 |
| LVSM+ MSE+LPIPS | 1000 | 17.67 | 0.612 | 0.450 | 0.54 | 1.15 |
At epoch 399 baseline long has 16.78 dB / 0.585, below both LVSM+ runs. With 4 sources the 1000-epoch models reach 19.03 dB (LVSM+) and 18.84 dB (baseline long).
Baseline vs. LVSM+ changes the decoder and capacity together, and each setting was run once. The MSE-only LVSM+ run was not extended, so the loss comparison is at 400 epochs, and the 1000-epoch LVSM+ run used a restarted learning-rate schedule. Detailed regions still show a faint 8-px grid pattern, and content the sources do not show stays blurry.
| Setting | Value |
|---|---|
| Config | Default configs/vggt.json, unchanged |
| Views / sampling range | 8 / 32 |
| Training | 1,000 epochs, batch size 16 |
| Optimizer | AdamW, lr 6×10−4, weight decay 0.01; 10-epoch warmup |
| Loss weights | 5 × camera + 1 × depth |
| Hardware | One RTX PRO 6000 (96 GB), 2 h 11 min |
| Metric | Val | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.151 | 0.164 |
| Depth δ1 ↑ | 0.799 | 0.775 |
| Rotation AUC @5° / 15° / 30° ↑ | 0.198 / 0.529 / 0.711 | 0.178 / 0.495 / 0.691 |
| Translation AUC @5° / 15° / 30° ↑ | 0.014 / 0.071 / 0.175 | 0.006 / 0.052 / 0.150 |
| Pose AUC @5° / 15° / 30° ↑ | 0.006 / 0.055 / 0.154 | 0.000 / 0.031 / 0.129 |
best.pt; full splits, 8 views from the first 32 frames, 224px, seed 825.
Each row: view 0's RGB and depth, ground truth, prediction.
09c1414f1b
21d970d8de
280b83fcf3
2b1dc6d6a5Depth recovers the room layout in every scene (δ1 0.80). Errors are at object boundaries and thin structures such as chair legs, which are smeared because depth is decoded from 14×14 patches.
Cameras: rotation is reasonable (AUC@30° 0.71), but translation direction is poor. The predicted cameras are bunched together, and because cameras in a 32-frame window are close to each other, a small position error gives a large direction error. This limits the joint pose AUC.
Val and test results are close, and training loss stays above validation loss, so the model underfits rather than overfits.