henrytsu · 16-825 · Assignment 4
After center cropping, intrinsics follow the half-pixel resize convention: f′ = s f, c′ = s(c + 0.5) − 0.5. For each pixel p = [u, v, 1], optical-axis depth gives X_camera = depth · K⁻¹p; the camera-to-world pose maps it to X_world = R X_camera + t.

All poses are expressed relative to the first sampled camera. Translations, depths and points are divided by ||mean(valid points)||, without subtracting the centroid. This fixes camera 0 at the origin and puts the valid-point centroid one unit away, giving samples a shared coordinate and scale convention.

The screenshot reports camera 0 at (0, 0, 0) and centroid distance 1.000.
A ray starts at the camera center. Its world direction is normalize(R K⁻¹p). The six Plücker channels are [d, o × d]: direction and moment, which jointly represent the oriented line. Training samples eight randomly ordered frames within a 32-frame window; the first sampled view sets the reference frame. Prepared validation/test samples use fixed views.
Four training samples passed the geometry check. Each uses eight 224 × 224 views; all centroid distances are 1.000.
| Sample | Point reprojection error (px) | Ray reprojection error (px) | Valid depth |
|---|---|---|---|
| e050c15a8d_frame_007080 | 0.000111 | 0.000108 | 97.79% |
| 5f99900f09_frame_002820 | 0.000123 | 0.000111 | 99.03% |
| 47b37eb6f9_frame_002720 | 0.000102 | 0.000089 | 96.10% |
| 4c5c60fa76_frame_006690 | 0.000109 | 0.000103 | 75.17% |
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011141905270051211,
"max_ray_reprojection_error_px": 0.00010764275066321716,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00010235906665911898,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Source tokens contain RGB and Plücker rays; target tokens contain only rays. Each target attends to the source views independently. The transformer decodes target tokens into RGB patches and learns with pixel MSE.
| Setting | Value |
|---|---|
| Train / validation / test scenes | 224 / 28 / 28 |
| Views / image size | 8 / 224 × 224 |
| Source views during training | 2 |
| Patch size / width / blocks / heads | 8 / 384 / 8 / 6 |
| Batch size / epochs | 4 / 100 |
| Optimizer | AdamW, lr 0.0002, weight decay 0.05, betas (0.9, 0.95) |
| Schedule | 200-step warmup, cosine decay to 10% of peak lr |
| Precision / gradient clipping / seed | bfloat16 / 1.0 / 825 |
| Hardware | NVIDIA RTX 5090, 32 GB |
After 2,000 steps on one fixed batch: PSNR 30.91 dB, SSIM 0.8697.

Columns: two source views, target ground truth, prediction.


Training visualization at epoch 96; columns follow the same source / GT / prediction order. Evaluation below uses the best validation checkpoint, epoch 100, on all 28 validation scenes. Views are selected from the first 32 prepared frames.
| Sources | Target-ray conditioning | PSNR (dB) | SSIM | Target images |
|---|---|---|---|---|
| 1 | correct | 15.21 | 0.5585 | 196 |
| 1 | shuffled | 11.81 | 0.4877 | 196 |
| 1 | none | 12.46 | 0.5292 | 196 |
| 2 | correct | 16.12 | 0.5601 | 168 |
| 2 | shuffled | 11.49 | 0.4724 | 168 |
| 2 | none | 12.43 | 0.5267 | 168 |
| 4 | correct | 16.85 | 0.5751 | 112 |
| 4 | shuffled | 11.58 | 0.4734 | 112 |
| 4 | none | 12.45 | 0.5362 | 112 |
With correct rays, PSNR rises from 15.21 to 16.85 dB as the source count increases from one to four. This comparison also changes the number and identity of target views. Shuffling target rays drops PSNR below 12 dB; removing them gives about 12.4 dB. The model uses camera conditioning, but held-out predictions remain blurry and less detailed than the overfit result.









Each row shows all source views, followed by target GT and prediction. Download LVSM metrics.
A frozen DINOv2 ViT-B/14 encoder provides image features. Eight pairs of frame/global attention blocks exchange information within and across views. Concatenated final-pair features feed camera and depth/confidence heads. Camera supervision uses translation, quaternion and FoV errors; depth supervision combines trimmed regression, confidence weighting and four-scale residual gradients.
| Setting | Value |
|---|---|
| Views / sampling range / image size | 8 / 32 / 224 × 224 |
| Width / attention heads / frame-global pairs | 384 / 6 / 8 |
| Batch size / epochs | 16 / 1,000 |
| Optimizer | AdamW, lr 0.0006, weight decay 0.01 |
| Schedule | 10-epoch warmup, cosine decay to 5% of peak lr |
| Camera / depth loss weights | 5 / 1 |
| Precision / gradient clipping / seed | bfloat16 / 1.0 / 825 |
One fixed eight-view training clip, 40 epochs × 25 steps, learning rate 0.001.
Depth AbsRel reaches 0.0376, δ1 0.9543, and mean rotation error 0.26°. The confidence term includes −0.2 log(confidence), so the combined depth loss can become negative.

| Ground truth | Prediction |
|---|---|
![]() |
![]() |

The best checkpoint is epoch 929, selected by validation loss. Both complete splits use eight views from the first 32 prepared frames, 224 px, seed 825; no batch limit. Each split has 28 scenes and 784 camera pairs. AUC values below are on a 0–1 scale.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1433 | 0.1577 |
| Depth δ1 ↑ | 0.8337 | 0.7986 |
| Rotation AUC @ 5° ↑ | 0.1829 | 0.1765 |
| Rotation AUC @ 15° ↑ | 0.5055 | 0.5042 |
| Rotation AUC @ 30° ↑ | 0.6951 | 0.7159 |
| Translation AUC @ 5° ↑ | 0.0099 | 0.0128 |
| Translation AUC @ 15° ↑ | 0.0707 | 0.0495 |
| Translation AUC @ 30° ↑ | 0.1853 | 0.1440 |
| Joint pose AUC @ 5° ↑ | 0.0041 | 0.0015 |
| Joint pose AUC @ 15° ↑ | 0.0574 | 0.0280 |
| Joint pose AUC @ 30° ↑ | 0.1654 | 0.1281 |
The held-out depth errors are larger than the overfit error. Rotation is more reliable than translation direction: validation AUC@30° is 0.6951 for rotation versus 0.1853 for translation. Translation limits joint-pose accuracy and can misalign clouds from different views. Test depth AbsRel rises to 0.1577; low joint-pose AUC shows that good local depth alone does not guarantee a consistent multi-view reconstruction.
Four validation scenes are shown below with matching GT/prediction camera viewpoints and axis ranges.
| Ground truth | Prediction |
|---|---|
![]() |
![]() |
| Ground truth | Prediction |
|---|---|
![]() |
![]() |
| Ground truth | Prediction |
|---|---|
![]() |
![]() |
| Ground truth | Prediction |
|---|---|
![]() |
![]() |