Multi-view transformers

16-825 Learning for 3D Vision · Assignment 4
Joshua Jacob · JJACOB2

1. Multi-view data preprocessing

The resize applies the half-pixel convention: u′ + 0.5 = sₓ(u + 0.5), with the analogous relation for v. This scales focal lengths, skew and principal points consistently with the image resize.

Depth is optical-axis z-depth. Unprojection uses Xcamera = zK⁻¹[u,v,1]ᵀ and Xworld = RXcamera + t. The unprojection vector is not normalized.

Canonicalization multiplies each pose by the inverse of the first sampled camera pose. It divides camera translations, depth and point coordinates by the distance from that first camera to the centroid of valid points. Camera 0 is at the origin, the centroid distance becomes one, and no centroid subtraction is applied. Invalid depth points remain zero.

World-space ray directions are RK⁻¹[u,v,1]ᵀ normalized to unit length; origins are camera centers. Plücker rays concatenate direction d and moment m = o × d. The moment describes the line’s offset from the origin and is unchanged when o moves along the ray.

1.2. Unprojection in meters

Unprojection — unprojection
Unprojection — unprojection
Unprojection — viewer screenshot
Unprojection — viewer screenshot

1.3. Canonicalized geometry

Canonicalization — canonicalization
Canonicalization — canonicalization
Canonicalization — viewer screenshot
Canonicalization — viewer screenshot

1.7. Data checks

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_005750",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9515330791473389,
      "max_reprojection_error_px": 0.00012680640793405473,
      "max_ray_reprojection_error_px": 0.00011620747682172805,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_003000",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9915946125984192,
      "max_reprojection_error_px": 9.918212890625e-05,
      "max_ray_reprojection_error_px": 9.918212890625e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_007760",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9745520949363708,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 7.82012939453125e-05,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "4c5c60fa76_frame_010770",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9885328412055969,
      "max_reprojection_error_px": 9.650505671743304e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Across 4 checked training samples, the largest point reprojection error is 0.000127 pixels and the largest ray reprojection error is 0.000116 pixels. Both are below the required 0.01-pixel tolerance.

Structured geometry-check results · Five-scene camera and centroid measurements

Across the four checked training samples, valid depth covers 95.15–99.16% of pixels. The maximum point reprojection error is 0.000127 pixels, and the largest ray reprojection error is 0.000116 pixels. All four centroid distances equal one within 0.00000006. The same sampled frames are shown before and after canonicalization.

2. Novel-view synthesis with Mini-LVSM

Source tokens encode RGB shifted to [−1,1] together with Plücker rays. Target tokens contain rays alone. Each target attends to all source tokens plus its own query tokens. Shared 2D sinusoidal positions locate the patches. The RGB head decodes target tokens using a sigmoid and inverse patchification. Training minimizes mean squared RGB error.

2.1. Fixed-batch overfitting

Training settingActual value
Device / precisionApple M3 Max (MPS) / fp32
Image size224 × 224
Views / sampling range8 / 32 frames
Batch size / seed1 / 825
Learning rate / weight decay0.0004 / 0.05
Training scenes1
Patch size / transformer width16 / 256
Transformer blocks / source views4 / 2
Fixed-batch optimization updates2000

The run uses a smaller Mini-LVSM than the supplied baseline (baseline: patch size 8, width 384 and eight blocks). The input/output geometry is retained. The reduced capacity limits conclusions about the quality attainable with the full baseline.

Fixed training batch metricValue
PSNR (dB)35.08048
SSIM0.92804
MSE0.00032

These metrics replay the memorized training batch; they do not measure held-out performance.

Fixed-batch overfit: source views, target ground truth, then predicted RGB.
Fixed-batch overfit: source views, target ground truth, then predicted RGB.

2.2. Training and evaluation

Training MSE and PSNR, with validation measured after each recorded epoch.
Training MSE and PSNR, with validation measured after each recorded epoch.

Recorded training epochs: 100. Total recorded training-plus-validation time: 27.31 minutes.

Validation selects epoch 94 (zero-based) by maximum PSNR. The GPU was shared with Mini-VGGT; recorded epoch times include validation and describe that concurrent workload.

Training settingActual value
Device / precisionApple M3 Max (MPS) / fp32
Image size224 × 224
Views / sampling range8 / 32 frames
Batch size / seed1 / 825
Learning rate / weight decay0.0004 / 0.05
Training scenes224
Patch size / transformer width16 / 256
Transformer blocks / source views4 / 2
Epochs completed / configured100 / 100

The run uses a smaller Mini-LVSM than the supplied baseline (baseline: patch size 8, width 384 and eight blocks). The input/output geometry is retained. The reduced capacity limits conclusions about the quality attainable with the full baseline.

The configured 100-epoch training schedule matches the course baseline; the completed count is reported above.

Actual run settings
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "batch_size": 1,
    "eval_num_frames": 32,
    "num_workers": 2,
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "seed": 825
  },
  "model": {
    "patch_size": 16,
    "d_model": 256,
    "n_layers": 4,
    "head_dim": 64,
    "mlp_ratio": 4.0,
    "use_qk_norm": true
  },
  "loss": {},
  "training": {
    "epochs": 100,
    "learning_rate": 0.0004,
    "weight_decay": 0.05,
    "betas": [
      0.9,
      0.95
    ],
    "warmup_steps": 100,
    "grad_clip": 1.0,
    "amp": true,
    "vis_every": 1,
    "output_dir": "outputs/lvsm"
  },
  "num_source_views": 2,
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/scene_splits.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 1,
    "num_workers": 2,
    "seed": 825
  }
}
Last saved validation visualization: source views, ground truth, prediction.
Last saved validation visualization: source views, ground truth, prediction.
SettingPSNR (dB)SSIMImages
S=1,conditioning=correct14.320880.54649196
S=1,conditioning=shuffled11.964610.50584196
S=1,conditioning=none12.440460.54156196
S=2,conditioning=correct14.809310.54334168
S=2,conditioning=shuffled11.841420.49889168
S=2,conditioning=none12.369530.53706168
S=4,conditioning=correct15.088740.55381112
S=4,conditioning=shuffled12.057150.50690112
S=4,conditioning=none12.370610.54639112

With 1 source views, correct-ray PSNR differs from shuffled rays by +2.356 dB and from removed rays by +1.880 dB.

With 2 source views, correct-ray PSNR differs from shuffled rays by +2.968 dB and from removed rays by +2.440 dB.

With 4 source views, correct-ray PSNR differs from shuffled rays by +3.032 dB and from removed rays by +2.718 dB.

Source-count comparisons also change which of the selected views serve as targets. They measure the supplied evaluation protocol, rather than a comparison with an identical target set. Ray-conditioning comparisons within one source count keep that target set fixed.

Validation S1_correct: source views, ground truth and prediction.
Validation S1_correct: source views, ground truth and prediction.
Validation S1_none: source views, ground truth and prediction.
Validation S1_none: source views, ground truth and prediction.
Validation S1_shuffled: source views, ground truth and prediction.
Validation S1_shuffled: source views, ground truth and prediction.
Validation S2_correct: source views, ground truth and prediction.
Validation S2_correct: source views, ground truth and prediction.
Validation S2_none: source views, ground truth and prediction.
Validation S2_none: source views, ground truth and prediction.
Validation S2_shuffled: source views, ground truth and prediction.
Validation S2_shuffled: source views, ground truth and prediction.
Validation S4_correct: source views, ground truth and prediction.
Validation S4_correct: source views, ground truth and prediction.
Validation S4_none: source views, ground truth and prediction.
Validation S4_none: source views, ground truth and prediction.
Validation S4_shuffled: source views, ground truth and prediction.
Validation S4_shuffled: source views, ground truth and prediction.

In the fixed-clip overfit example, the predictions preserve the desk, monitors, yellow chairs and the changes in viewpoint across targets. Fine edges retain a visible grain or patch texture, particularly on the floor and walls. This is a successful memorization check; it does not establish generalization to new rooms.

With correct rays, PSNR increases from 14.321 dB with one source view to 14.809 dB with two and 15.089 dB with four. SSIM is 0.54649, 0.54334 and 0.55381, respectively, so structural similarity is not monotonic. These source-count comparisons also change the target subset, as noted above; they do not isolate source count with identical targets.

Correct target rays outperform shuffled rays by 2.356–3.032 dB and zero rays by 1.880–2.718 dB across the three source counts. This indicates that the model uses target-camera conditioning. Shuffled rays are worse than zero rays in both PSNR and SSIM: an incorrect view query is more damaging here than an absent query. Zero-ray SSIM remains fairly close to correct-ray SSIM despite its lower PSNR, so SSIM alone understates the difference.

On the held-out dining-room example, the predictions reproduce broad beige and brown regions and a bright window area, but they do not resolve the table, chair legs, shelving or the recliner. Patch boundaries remain visible. Four source views improve the aggregate metrics without restoring those details. The gap from the fixed-clip overfit result shows that successful memorization does not imply strong generalization; reduced model capacity and optimization settings are possible contributors, but this run does not isolate their effects.

3. Multi-view geometry with Mini-VGGT

A frozen DINO encoder extracts image patches. Alternating attention first mixes tokens within each view and then across all views. The final frame and global features feed the camera and dense depth/confidence heads. Camera supervision uses L1 errors on translation, quaternion encoding and field of view; the depth objective combines confidence-weighted regression and spatial gradient terms.

3.1. Fixed-clip overfitting

3.1. Fixed-clip overfitting: total, camera and depth loss.
3.1. Fixed-clip overfitting: total, camera and depth loss.
Fixed-clip metricInitialFinal recorded epoch
Total loss5.411870.09692
Camera loss0.343150.00077
Depth loss3.696140.09305
Depth AbsRel0.605860.06653
Depth δ10.200030.91741

All metrics in this subsection replay the fixed training clip. The normal-run results below evaluate held-out scenes.

Training settingActual value
Device / precisionApple M3 Max (MPS) / fp32
Image size224 × 224
Views / sampling range8 / 32 frames
Batch size / seed1 / 825
Learning rate / weight decay0.001 / 0.01
Training scenes1
Frozen encoderdinov2_vits14
Transformer width / frame-global pairs192 / 3
Epochs completed / configured20 / 20
Updates per fixed-clip epoch50

This run reduces the supplied model’s capacity: the baseline uses DINOv2 ViT-B/14, width 384 and eight frame/global attention pairs. The eight-view, 32-frame and 224-pixel geometry settings are retained. Recorded epochs describe the actual short training run, rather than the baseline’s 1,000-epoch schedule.

Last recorded epoch: 20. Recorded optimization epochs: 20.

Actual run settings
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "num_views": 8,
    "sampling_range": 32,
    "eval_num_frames": 32,
    "batch_size": 1,
    "num_workers": 2
  },
  "model": {
    "embed_dim": 192,
    "num_heads": 6,
    "num_layers": 3,
    "predict_points": false,
    "encoder_name": "dinov2_vits14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 20,
    "learning_rate": 0.001,
    "weight_decay": 0.01,
    "output_dir": "outputs/vggt_overfit",
    "overfit_scenes": 1,
    "overfit_steps_per_epoch": 50,
    "min_lr_ratio": 0.05,
    "log_every": 10,
    "warmup_epochs": 0,
    "overfit_epochs": 20,
    "overfit_learning_rate": 0.001
  },
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/vggt_overfit_split.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 1,
    "num_workers": 2,
    "seed": 825
  },
  "run": {
    "overfit": true,
    "overfit_scenes": 1,
    "steps_per_epoch": 50,
    "max_val_batches": null,
    "world_size": 1,
    "precision": "fp32"
  }
}
Ground truth and predicted geometry — scene 00.
Ground truth and predicted geometry — scene 00.
Interactive viewer: ground-truth geometry — viewer ground truth.
Interactive viewer: ground-truth geometry — viewer ground truth.
Interactive viewer: predicted cameras and depth — viewer prediction.
Interactive viewer: predicted cameras and depth — viewer prediction.

3.2. Normal training and held-out evaluation

3.2. Normal training and held-out evaluation: total, camera and depth loss.
3.2. Normal training and held-out evaluation: total, camera and depth loss.
Training settingActual value
Device / precisionApple M3 Max (MPS) / fp32
Image size224 × 224
Views / sampling range8 / 32 frames
Batch size / seed1 / 825
Learning rate / weight decay0.0006 / 0.01
Training scenes224
Frozen encoderdinov2_vits14
Transformer width / frame-global pairs192 / 3
Epochs completed / configured20 / 40
Loss evaluation devicecpu
Checkpoint-safe epoch cap20

This run reduces the supplied model’s capacity: the baseline uses DINOv2 ViT-B/14, width 384 and eight frame/global attention pairs. The eight-view, 32-frame and 224-pixel geometry settings are retained. Recorded epochs describe the actual short training run, rather than the baseline’s 1,000-epoch schedule.

Last recorded epoch: 20. Recorded optimization epochs: 20.

Validation selects epoch 18 by minimum combined camera/depth loss. Recorded epoch times include the full validation pass and exclude checkpoint writes and curve plotting; MPS was shared with Mini-LVSM.

Training resumed from the saved epoch-7 checkpoint with its optimizer state and original 40-epoch learning-rate schedule. The differentiable loss calculation moved from MPS to CPU to avoid variable-shape loss overhead; predictions still come from the MPS model. The objective and its weights are unchanged.

The interrupted, unsaved portion of epoch 8 was discarded. Its final logged step was 40; the number of additional unlogged updates is unknown. The supplied resume loader resets its sampling seed and data order, so this run does not preserve the uninterrupted loader RNG trajectory.

Actual run settings
{
  "data_config": "configs/scannetpp/iphone/data.json",
  "data_overrides": {
    "num_views": 8,
    "sampling_range": 32,
    "eval_num_frames": 32,
    "batch_size": 1,
    "num_workers": 2
  },
  "model": {
    "embed_dim": 192,
    "num_heads": 6,
    "num_layers": 3,
    "predict_points": false,
    "encoder_name": "dinov2_vits14",
    "image_size": 224
  },
  "loss": {
    "camera": 5.0,
    "depth": 1.0
  },
  "training": {
    "epochs": 40,
    "learning_rate": 0.0006,
    "weight_decay": 0.01,
    "output_dir": "outputs/vggt_train",
    "overfit_scenes": 1,
    "overfit_steps_per_epoch": 50,
    "min_lr_ratio": 0.05,
    "log_every": 10,
    "warmup_epochs": 0,
    "overfit_epochs": 20,
    "overfit_learning_rate": 0.001
  },
  "data": {
    "root": "data/scannetpp/processed",
    "eval_root": "data/scannetpp/prepared_eval",
    "eval_num_frames": 32,
    "split_file": "configs/scannetpp/iphone/scene_splits.json",
    "num_views": 8,
    "sampling_range": 32,
    "image_size": 224,
    "batch_size": 1,
    "num_workers": 2,
    "seed": 825
  },
  "run": {
    "overfit": false,
    "overfit_scenes": 1,
    "steps_per_epoch": null,
    "max_val_batches": null,
    "world_size": 1,
    "precision": "fp32",
    "stop_after_epoch": 20,
    "loss_device": "cpu"
  }
}
Ground truth and predicted geometry — scene 00.
Ground truth and predicted geometry — scene 00.
Ground truth and predicted geometry — scene 01.
Ground truth and predicted geometry — scene 01.
Ground truth and predicted geometry — scene 02.
Ground truth and predicted geometry — scene 02.
Ground truth and predicted geometry — scene 03.
Ground truth and predicted geometry — scene 03.

Validation raw metrics

Test raw metrics

MetricValidationTest
Depth AbsRel0.184100.21100
Depth δ10.722690.66019
Rotation AUC @5°0.097450.11709
Rotation AUC @15°0.249490.29277
Rotation AUC @30°0.422660.47997
Translation AUC @5°0.004590.00077
Translation AUC @15°0.027640.02551
Translation AUC @30°0.084310.09252
Joint pose AUC @5°0.000770.00026
Joint pose AUC @15°0.014120.01497
Joint pose AUC @30°0.068410.07058
SplitScenesAll camera pairsPairs with undefined translation (subset)
Validation287840
Test287840

Depth metrics include all valid ground-truth pixels. Rotation AUC uses all camera pairs. Translation and joint-pose AUC omit pairs whose ground-truth translation direction is undefined and allow translation sign ambiguity.

On the fixed training clip, the predicted cloud recovers the room layout, floor and camera trajectory. The predicted surfaces are somewhat thicker and less crisp than the measured depth cloud. Both panels use the same coordinates, axis limits, RGB colors and GT-valid pixels; no alignment is fitted to the prediction.

The four held-out scenes show substantially larger errors than the memorized clip. In 09c1414f1b, the predicted cloud retains broad room regions but thickens and mixes surfaces. In 21d970d8de, it compresses the extent of the reconstructed structure. In 280b83fcf3, the tall ground-truth structure becomes a shorter, denser cloud and its separated surfaces are less distinct. In 2b1dc6d6a5, planar surfaces become diffuse, with scattered points beyond the ground-truth structure.

Camera prediction is a major limitation: predicted camera centers cluster near the reference origin, while the ground-truth trajectories span larger baselines, particularly in the fourth scene. Errors in both camera poses and per-view depth therefore accumulate when the views are fused into a point cloud. The comparison uses the same canonical coordinates, axis limits, sampled pixels and RGB colors for GT and prediction, with no fitted alignment. Recognizable colors do not by themselves establish accurate geometry.

Validation depth AbsRel is 0.18410 with δ1 = 0.72269; test AbsRel is 0.21100 with δ1 = 0.66019. Test depth accuracy is lower on both measures. Joint pose AUC at 30° is only 0.06841 on validation and 0.07058 on test, consistent with the inaccurate camera trajectories in the comparisons. The higher test rotation AUC does not remove the translation-direction errors.

The normal run covers all 224 training scenes but stops after 20 epochs of a 40-epoch schedule and uses a smaller encoder and transformer than the supplied baseline. Its weak held-out pose accuracy contrasts with the successful overfit check. Longer training or greater capacity may improve generalization, but neither explanation is isolated by these experiments.

Artifact status

All required artifact categories are present. Measurements and settings are reported below.

Machine-readable artifact status · Recorded results

Reproduction

Setup and experiment commands · Original assignment overview

python build_report.py
python verify_artifacts.py
python package_submission.py

Starter code and task definitions: learning3d/assignment4_f26. Training and evaluation use the course ScanNet++ iPhone split. This report displays saved outputs from the implementation; the supplied TA reference is not a submission result.

Recorded environment · Package versions

Implementation checks

23 passed in 4.08s. Geometry checks cover resize and camera conventions, canonicalization and reprojection. Model checks cover patch/token construction, output shapes, gradients and loss behavior.

Saved test output

python -m pytest -q tests