Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

Canonicalization: Express cameras and points in the first sampled camera's frame using T0⁻¹ Ti. Divide translations, depth and points by s = ‖mean(pvalid)‖, without subtracting the centroid. Camera 0 is at the origin and the valid-point centroid has distance one.

World-space rays: The origin o is the camera center. Rotate K⁻¹[u,v,1]ᵀ into world coordinates and normalize it to obtain direction d. Depth unprojection uses optical-axis z-depth, not distance along a unit ray.

Plücker moment: Each ray is [d, o × d]. The moment describes the line's offset and is unchanged when the origin moves along the ray.

Unprojection and canonicalization

Training geometry: unprojection. Original world coordinates. First camera center (X, Y, Z): (3.3463, 4.6765, 1.4680) m. Valid-point centroid distance: 6.668 m.
Training geometry: unprojection. Original world coordinates. First camera center (X, Y, Z): (3.3463, 4.6765, 1.4680) m. Valid-point centroid distance: 6.668 m.
Training geometry: canonicalization. Canonical first-view coordinates. First camera center (X, Y, Z): (0.0000, 0.0000, 0.0000) normalized units. Scene scale: 1.867 m. Valid-point centroid distance: 1.000 normalized units.
Training geometry: canonicalization. Canonical first-view coordinates. First camera center (X, Y, Z): (0.0000, 0.0000, 0.0000) normalized units. Scene scale: 1.867 m. Valid-point centroid distance: 1.000 normalized units.

check_data.py output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001089862416847609,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

Q2.1 — Overfitting

After 2,000 steps on one fixed batch: 30.9210 dB PSNR.

Overfit result. Columns: source 1, source 2, ground truth, prediction.
Overfit result. Columns: source 1, source 2, ground truth, prediction.

Q2.2 — Training and evaluation

Settings: 100 epochs on 224 training scenes; 8 views, 2 sources, batch size 4, 224px, seed 825. Patch size 8, width 384, 8 transformer blocks. AdamW, learning rate 0.0002, weight decay 0.05, 200 warmup steps, cosine decay, gradient clipping 1.0 and bfloat16 training with float32 MSE loss. Validation uses 28 scenes and eight views from the first 32 prepared frames. Best validation PSNR is 16.1162 dB at epoch 100.

Training and validation curves.
Training and validation curves.
Validation visualization during training, epoch 96. Columns: source 1, source 2, ground truth, prediction.
Validation visualization during training, epoch 96. Columns: source 1, source 2, ground truth, prediction.

Validation ablations

All settings use the same best checkpoint. Correct uses true target rays; shuffled uses another target view's rays; none uses zero target rays. Higher PSNR and SSIM are better.

SourcesTarget raysPSNR (dB)SSIM
1correct15.19670.55785
1shuffled11.80190.48694
1none12.46830.52908
2correct16.11620.55987
2shuffled11.48150.47194
2none12.43880.52679
4correct16.85720.57494
4shuffled11.58930.47337
4none12.45620.53640

With correct rays, PSNR increases from 15.20 to 16.12 to 16.86 dB as source count increases. At two sources, shuffling rays reduces PSNR from 16.12 to 11.48 dB; removing them gives 12.44 dB. This indicates that the model uses camera conditioning. More sources provide additional scene information, though source-count comparisons also change the target set and test source counts different from training. Predictions capture broad colors and layout but remain blurred, especially at object boundaries and fine textures. These single-seed results do not establish robust geometric extrapolation.

1 source views: evaluation images

1 sources, correct target rays. Columns: source views, ground truth, prediction.
1 sources, correct target rays. Columns: source views, ground truth, prediction.
1 sources, shuffled target rays. Columns: source views, ground truth, prediction.
1 sources, shuffled target rays. Columns: source views, ground truth, prediction.
1 sources, none target rays. Columns: source views, ground truth, prediction.
1 sources, none target rays. Columns: source views, ground truth, prediction.

2 source views: evaluation images

2 sources, correct target rays. Columns: source views, ground truth, prediction.
2 sources, correct target rays. Columns: source views, ground truth, prediction.
2 sources, shuffled target rays. Columns: source views, ground truth, prediction.
2 sources, shuffled target rays. Columns: source views, ground truth, prediction.
2 sources, none target rays. Columns: source views, ground truth, prediction.
2 sources, none target rays. Columns: source views, ground truth, prediction.

4 source views: evaluation images

4 sources, correct target rays. Columns: source views, ground truth, prediction.
4 sources, correct target rays. Columns: source views, ground truth, prediction.
4 sources, shuffled target rays. Columns: source views, ground truth, prediction.
4 sources, shuffled target rays. Columns: source views, ground truth, prediction.
4 sources, none target rays. Columns: source views, ground truth, prediction.
4 sources, none target rays. Columns: source views, ground truth, prediction.

3. Multi-View Geometry with Mini-VGGT

Q3.1 — Overfitting

One fixed training clip, 40 epochs × 25 steps, learning rate 0.001. Final depth AbsRel 0.03565, δ1 0.95710, and joint pose AUC@30° 0.99762.

Overfit total, camera and depth losses. Component panels are unweighted.
Overfit total, camera and depth losses. Component panels are unweighted.
Fixed training clip: ground-truth geometry.
Fixed training clip: ground-truth geometry.
Fixed training clip: predicted geometry.
Fixed training clip: predicted geometry.

Q3.2 — Training and evaluation

Settings: frozen DINOv2 ViT-B/14, eight frame/global attention pairs, width 384, six heads; 1000 epochs on 224 training scenes, batch size 16, 8 views, sampling range 32, 224px, seed 825. AdamW, learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine decay to 5% of peak learning rate, gradient clipping 1.0 and bfloat16 attention. Best checkpoint: epoch 850, selected by validation loss.

Normal-training and validation total, camera and depth losses. Component panels are unweighted.
Normal-training and validation total, camera and depth losses. Component panels are unweighted.

Validation and test each use all 28 scenes, eight views from the first 32 prepared frames, 224px and seed 825, with no batch limit. Lower AbsRel is better; higher δ1 and AUC are better.

MetricValidationTest
Depth AbsRel0.145810.15223
Depth δ10.827130.82036
Rotation AUC@5°0.169640.20102
Rotation AUC@15°0.503740.54685
Rotation AUC@30°0.705230.74949
Translation AUC@5°0.021940.02066
Translation AUC@15°0.092520.09379
Translation AUC@30°0.196980.22976
Joint pose AUC@5°0.002810.00408
Joint pose AUC@15°0.067180.06463
Joint pose AUC@30°0.176660.20557

Held-out geometry and errors

Predicted depth captures broad scene structure, but camera errors can misalign surfaces across views. Among these four scenes, 280b83fcf3 has the highest depth AbsRel and mean rotation error. Translation errors remain substantial, consistent with the lower translation and joint pose AUCs. Depth and relative-pose accuracy measure different aspects of reconstruction; accurate per-view depth does not guarantee aligned point clouds.

Scene 1: 09c1414f1b_frame_000000

Depth AbsRel 0.0873, δ1 0.9412; mean relative rotation error 4.68°, translation-direction error 27.33°; joint pose AUC@30° 0.3655.

Ground-truth geometry.
Ground-truth geometry.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.

Scene 2: 21d970d8de_frame_000000

Depth AbsRel 0.0593, δ1 0.9648; mean relative rotation error 3.84°, translation-direction error 44.83°; joint pose AUC@30° 0.0679.

Ground-truth geometry.
Ground-truth geometry.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.

Scene 3: 280b83fcf3_frame_000000

Depth AbsRel 0.1064, δ1 0.8924; mean relative rotation error 13.47°, translation-direction error 42.03°; joint pose AUC@30° 0.2012.

Ground-truth geometry.
Ground-truth geometry.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.

Scene 4: 2b1dc6d6a5_frame_000000

Depth AbsRel 0.0789, δ1 0.9713; mean relative rotation error 6.72°, translation-direction error 10.31°; joint pose AUC@30° 0.5881.

Ground-truth geometry.
Ground-truth geometry.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.
Predicted geometry. Same coordinates and axis ranges as ground truth, without alignment.

Q3.3 — Number of input views

Question: how do 2, 4 and 8 input views affect geometry accuracy, inference time and GPU memory?

Settings: the same epoch-850 checkpoint, trained with eight views, is evaluated on all 28 validation scenes. Inputs are nested subsets of eight frames sampled from the first 32 frames (seed 825). Each setting scores the same two views and camera pair, with normalization fixed. No retraining or alignment is used. Batch size 1, 224px, NVIDIA L4, bfloat16.

Latency is the median synchronized forward time after three warmups and ten repetitions per scene/setting, excluding data loading, transfer and metrics. Memory is peak allocated tensor memory, including the model, rather than total GPU process memory.

Input viewsAbsRel ↓δ1 ↑Rotation AUC@30° ↑ Translation AUC@30° ↑Joint pose AUC@30° ↑Latency (ms)Peak memory (MiB)
20.198060.658550.679760.213100.2023827.31590.2
40.153410.797860.703570.192860.1773827.09606.5
80.142280.822140.694050.223810.2083327.47641.5
Accuracy, latency and memory versus input views. Latency bars show the 10th–90th percentiles.
Accuracy, latency and memory versus input views. Latency bars show the 10th–90th percentiles.
First four validation scenes: common-view RGB, GT depth and predictions with 2/4/8 inputs. Depth colors and GT-valid masks are shared within each row.
First four validation scenes: common-view RGB, GT depth and predictions with 2/4/8 inputs. Depth colors and GT-valid masks are shared within each row.

Conclusion: eight views reduce AbsRel from 0.1981 to 0.1423, improving depth in 21/28 scenes. Pose accuracy is not monotonic. Latency remains near 27 ms with overlapping timing ranges, while memory rises from 590 to 641 MiB. Extra context can help resolve scene structure, but redundant views and occlusion can limit pose gains. Shared weights and fixed dispatch costs may explain the modest memory increase and similar latency; thin structures remain blurred.

Limitations: one checkpoint, one frame selection per scene, and only two scored views/one camera pair. Fewer views differ from training, and no models were retrained. Results do not establish training-view-count effects; timing is specific to this GPU and batch size.