Assignment 4: Multi-View Transformers

junchif

1. Multi-View Data Preprocessing

Canonicalization transforms cameras and depth-unprojected points into the first sampled camera frame, then divides camera translations, depth and points by the distance from that camera to the valid-point centroid. The centroid is not subtracted. Camera 0 is identity and the normalized centroid distance is 1. World-space ray directions are normalize(R K⁻¹[u,v,1]), with camera-center origins. The Plücker moment m = o × d describes the ray line’s offset from the origin; channels are [d,m].

036bce3393_frame_004470 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
036bce3393_frame_004470 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
076c822ecc_frame_002890 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
076c822ecc_frame_002890 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
079a326597_frame_007430 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
079a326597_frame_007430 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
07f5b601ee_frame_003630 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
07f5b601ee_frame_003630 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
07ff1c45bb_frame_000460 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
07ff1c45bb_frame_000460 — unprojection (left) and canonicalization (right). Camera 0 and centroid distance are shown in each panel.
check_data.py output — four training samples
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_005750",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9515330791473389,
      "max_reprojection_error_px": 0.00012537825386971235,
      "max_ray_reprojection_error_px": 0.00011620747682172805,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_003000",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9915946125984192,
      "max_reprojection_error_px": 0.00010850114631466568,
      "max_ray_reprojection_error_px": 9.650505671743304e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_007760",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9745520949363708,
      "max_reprojection_error_px": 9.770392352947965e-05,
      "max_ray_reprojection_error_px": 7.912623550510034e-05,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "4c5c60fa76_frame_010770",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9885328412055969,
      "max_reprojection_error_px": 8.897326188161969e-05,
      "max_ray_reprojection_error_px": 8.217109279939905e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      1,
      8,
      3,
      224,
      224
    ],
    "depth": [
      1,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      1,
      8,
      1,
      224,
      224
    ],
    "K": [
      1,
      8,
      3,
      3
    ],
    "frame_numbers": [
      1,
      8
    ],
    "T_world_from_camera": [
      1,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      1,
      4,
      4
    ],
    "scene_scale": [
      1
    ],
    "points_world": [
      1,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      1,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      1,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      1,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1. Overfitting

One fixed batch, 2000 optimization steps. Final overfit PSNR: 36.59 dB.

Overfit visualization. Columns: source views, target ground truth, prediction.
Overfit visualization. Columns: source views, target ground truth, prediction.

2.2. Training and Evaluation

Training settingValue
Input224×224; 8 views; sampling range 32; 2 sources and 6 targets
ModelPatch 8; width 384; 8 transformer blocks; attention head width 64
OptimizerAdamW; learning rate 0.0002; weight decay 0.05; betas (0.9,0.95)
Schedule100 epochs; batch size 2; warmup 200 steps; cosine decay; gradient clip 1
Precision and seedBF16 forward, FP32 loss; seed 825
Training and validation curves. Training PSNR averages predictions during optimization; validation uses the end-of-epoch model.
Training and validation curves. Training PSNR averages predictions during optimization; validation uses the end-of-epoch model.
Saved training visualization at epoch 96. Columns: source views, target ground truth, prediction.
Saved training visualization at epoch 96. Columns: source views, target ground truth, prediction.
Source viewsTarget raysPSNR ↑SSIM ↑
1correct15.070.5576
1shuffled11.830.4915
1none12.420.5379
2correct16.030.5591
2shuffled11.530.4764
2none12.410.5356
4correct16.660.5733
4shuffled11.650.4785
4none12.450.5467
Evaluation visualization — 1 source views, correct target rays
1 source views, correct target rays. Columns: sources, ground truth, prediction.
1 source views, correct target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 1 source views, shuffled target rays
1 source views, shuffled target rays. Columns: sources, ground truth, prediction.
1 source views, shuffled target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 1 source views, none target rays
1 source views, none target rays. Columns: sources, ground truth, prediction.
1 source views, none target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 2 source views, correct target rays
2 source views, correct target rays. Columns: sources, ground truth, prediction.
2 source views, correct target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 2 source views, shuffled target rays
2 source views, shuffled target rays. Columns: sources, ground truth, prediction.
2 source views, shuffled target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 2 source views, none target rays
2 source views, none target rays. Columns: sources, ground truth, prediction.
2 source views, none target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 4 source views, correct target rays
4 source views, correct target rays. Columns: sources, ground truth, prediction.
4 source views, correct target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 4 source views, shuffled target rays
4 source views, shuffled target rays. Columns: sources, ground truth, prediction.
4 source views, shuffled target rays. Columns: sources, ground truth, prediction.
Evaluation visualization — 4 source views, none target rays
4 source views, none target rays. Columns: sources, ground truth, prediction.
4 source views, none target rays. Columns: sources, ground truth, prediction.

With correct rays, PSNR increases from 15.07 to 16.03 to 16.66 dB for 1, 2 and 4 sources. The prescribed source/target partition also changes the target set, so this is not a fixed-target causal comparison. At two sources, shuffled and zero target rays reduce PSNR by 4.50 and 3.62 dB, respectively, supporting the importance of target-camera conditioning. Predictions retain broad color and light regions but blur chair legs, shelf contents and window edges.

2.3. Bonus: MSE + L1 Loss

Question: does adding L1 improve image quality? Compare MSE with MSE + 0.1 L1, using the same architecture, initial seed, sampling settings, optimizer and 100-epoch budget.

Sources (correct rays)MSE PSNRMSE+L1 PSNRMSE SSIMMSE+L1 SSIM
115.0715.120.55760.5585
216.0315.990.55910.5583
416.6616.590.57330.5725
MSE baseline, two sources and correct target rays.
MSE baseline, two sources and correct target rays.
MSE + 0.1 L1, the same source and target images.
MSE + 0.1 L1, the same source and target images.

At two sources, L1 changes PSNR by -0.036 dB and SSIM by -0.0008. There is no consistent metric improvement across source counts. Both example predictions remain strongly blurred, without a convincing sharpness gain. L1 changes the pixel-error penalty but does not resolve this model’s missing fine detail. This is a single-seed experiment; the L1 run resumed after an interruption, preserving model/optimizer state but reseeding subsequent sample order, so the comparison is not perfectly matched in clip order.

3. Multi-View Geometry with Mini-VGGT

3.1. Overfitting

One fixed training clip; 40 epochs × 25 steps, batch size 1 and learning rate 0.001.

Fixed-clip total, camera and depth losses. The confidence term includes −0.2 log(confidence), so the depth and total objectives may become negative.
Fixed-clip total, camera and depth losses. The confidence term includes −0.2 log(confidence), so the depth and total objectives may become negative.
Overfit GT/prediction comparison. Shared canonical axes, no GT alignment; red markers show camera centers.
Overfit GT/prediction comparison. Shared canonical axes, no GT alignment; red markers show camera centers.

3.2. Training and Evaluation

Training settingValue
Input224×224; 8 views; sampling range 32
ModelFrozen DINOv2 ViT-B/14; width 384; 8 frame/global pairs; 6 heads; camera and depth heads
OptimizerAdamW; base learning rate 0.0006; weight decay 0.01; gradient clip 1
Budget200 total epochs; batch size 4; seed 825; BF16 forward
Learning-rate schedule10-epoch warmup; original 1000-epoch plan through epoch 108; resumed with AdamW state and continuous cosine decay from 0.0005863292 to 0.00003 over epochs 109–200
Loss weightsCamera 5; depth 1
EvaluationFull validation/test splits; 8 views from first 32 prepared frames; 224px; seed 825
Full-training total, camera and depth loss curves.
Full-training total, camera and depth loss curves.
SplitDepth AbsRel ↓Depth δ1 ↑ThresholdRotation AUC ↑Translation AUC ↑Joint AUC ↑
Validation0.16470.78275°0.14490.00770.0003
Validation0.16470.782715°0.38980.04940.0281
Validation0.16470.782730°0.57730.12680.0999
Test0.18330.75145°0.16840.00590.0026
Test0.18330.751415°0.43130.03940.0274
Test0.18330.751430°0.63720.11690.0990
09c1414f1b_frame_000000 — validation GT and prediction, shared axes without GT alignment.
09c1414f1b_frame_000000 — validation GT and prediction, shared axes without GT alignment.
21d970d8de_frame_000000 — validation GT and prediction, shared axes without GT alignment.
21d970d8de_frame_000000 — validation GT and prediction, shared axes without GT alignment.
280b83fcf3_frame_000000 — validation GT and prediction, shared axes without GT alignment.
280b83fcf3_frame_000000 — validation GT and prediction, shared axes without GT alignment.
2b1dc6d6a5_frame_000000 — validation GT and prediction, shared axes without GT alignment.
2b1dc6d6a5_frame_000000 — validation GT and prediction, shared axes without GT alignment.

Coarse room structure is recovered in 09c1414f1b, but predicted surfaces are thicker and less consistent. In 2b1dc6d6a5, points spread over a larger region than GT and camera-center arrangement differs. Depth AbsRel is 0.1647 on validation and 0.1833 on test, while mean translation-direction errors remain about 49° and 47°. Low joint-pose AUC indicates unreliable camera recovery despite plausible coarse depth. The 200-epoch resumed budget and the gap between fixed-clip overfitting and held-out scenes limit this result.

3.3. Bonus: Number of Input Views

Question: how does input-view count affect accuracy, runtime and memory? Evaluate the same selected checkpoint with 2, 4 and 8 views, using the first 32 prepared validation frames, seed 825, 224px and batch size 1. No additional training is used. Inference timing includes input transfer, excludes an untimed warmup and synchronizes CUDA; memory is PyTorch peak allocated memory during evaluation.

ViewsDepth AbsRel ↓Depth δ1 ↑Joint AUC@30° ↑Inference ms/clip ↓Peak allocated GiB ↓
20.13020.82590.102442.150.691
40.14760.80650.116941.910.706
80.16470.78270.099956.870.755
Validation view-count experiment: accuracy, inference time and evaluation tensor memory.
Validation view-count experiment: accuracy, inference time and evaluation tensor memory.

Eight views use more memory and take longer than two or four views. The small two-versus-four timing difference is not persuasive evidence of a speed improvement in a single measurement. Fewer views have lower reported depth error here, but view selection, scene normalization and scored camera pairs also change (28, 168 and 784 pairs), so this does not establish that fewer views intrinsically improve reconstruction. Joint-pose accuracy remains weak for all counts. A fixed-frame/fixed-pair experiment and repeated timings would better isolate context size and runtime variability.