1. Multi-view data preprocessing
The resize applies the half-pixel convention: u′ + 0.5 = sₓ(u + 0.5), with the analogous relation for v. This scales focal lengths, skew and principal points consistently with the image resize.
Depth is optical-axis z-depth. Unprojection uses Xcamera = zK⁻¹[u,v,1]ᵀ and Xworld = RXcamera + t. The unprojection vector is not normalized.
Canonicalization multiplies each pose by the inverse of the first sampled camera pose. It divides camera translations, depth and point coordinates by the distance from that first camera to the centroid of valid points. Camera 0 is at the origin, the centroid distance becomes one, and no centroid subtraction is applied. Invalid depth points remain zero.
World-space ray directions are RK⁻¹[u,v,1]ᵀ normalized to unit length; origins are camera centers. Plücker rays concatenate direction d and moment m = o × d. The moment describes the line’s offset from the origin and is unchanged when o moves along the ray.
1.2. Unprojection in meters


1.3. Canonicalized geometry


1.7. Data checks
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_005750",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9515330791473389,
"max_reprojection_error_px": 0.00012680640793405473,
"max_ray_reprojection_error_px": 0.00011620747682172805,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_003000",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9915946125984192,
"max_reprojection_error_px": 9.918212890625e-05,
"max_ray_reprojection_error_px": 9.918212890625e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_007760",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9745520949363708,
"max_reprojection_error_px": 0.00010235906665911898,
"max_ray_reprojection_error_px": 7.82012939453125e-05,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "4c5c60fa76_frame_010770",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9885328412055969,
"max_reprojection_error_px": 9.650505671743304e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Across 4 checked training samples, the largest point reprojection error is 0.000127 pixels and the largest ray reprojection error is 0.000116 pixels. Both are below the required 0.01-pixel tolerance.
Structured geometry-check results · Five-scene camera and centroid measurements
Across the four checked training samples, valid depth covers 95.15–99.16% of pixels. The maximum point reprojection error is 0.000127 pixels, and the largest ray reprojection error is 0.000116 pixels. All four centroid distances equal one within 0.00000006. The same sampled frames are shown before and after canonicalization.
2. Novel-view synthesis with Mini-LVSM
Source tokens encode RGB shifted to [−1,1] together with Plücker rays. Target tokens contain rays alone. Each target attends to all source tokens plus its own query tokens. Shared 2D sinusoidal positions locate the patches. The RGB head decodes target tokens using a sigmoid and inverse patchification. Training minimizes mean squared RGB error.
2.1. Fixed-batch overfitting
| Training setting | Actual value |
|---|---|
| Device / precision | Apple M3 Max (MPS) / fp32 |
| Image size | 224 × 224 |
| Views / sampling range | 8 / 32 frames |
| Batch size / seed | 1 / 825 |
| Learning rate / weight decay | 0.0004 / 0.05 |
| Training scenes | 1 |
| Patch size / transformer width | 16 / 256 |
| Transformer blocks / source views | 4 / 2 |
| Fixed-batch optimization updates | 2000 |
The run uses a smaller Mini-LVSM than the supplied baseline (baseline: patch size 8, width 384 and eight blocks). The input/output geometry is retained. The reduced capacity limits conclusions about the quality attainable with the full baseline.
| Fixed training batch metric | Value |
|---|---|
| PSNR (dB) | 35.08048 |
| SSIM | 0.92804 |
| MSE | 0.00032 |
These metrics replay the memorized training batch; they do not measure held-out performance.

2.2. Training and evaluation

Recorded training epochs: 100. Total recorded training-plus-validation time: 27.31 minutes.
Validation selects epoch 94 (zero-based) by maximum PSNR. The GPU was shared with Mini-VGGT; recorded epoch times include validation and describe that concurrent workload.
| Training setting | Actual value |
|---|---|
| Device / precision | Apple M3 Max (MPS) / fp32 |
| Image size | 224 × 224 |
| Views / sampling range | 8 / 32 frames |
| Batch size / seed | 1 / 825 |
| Learning rate / weight decay | 0.0004 / 0.05 |
| Training scenes | 224 |
| Patch size / transformer width | 16 / 256 |
| Transformer blocks / source views | 4 / 2 |
| Epochs completed / configured | 100 / 100 |
The run uses a smaller Mini-LVSM than the supplied baseline (baseline: patch size 8, width 384 and eight blocks). The input/output geometry is retained. The reduced capacity limits conclusions about the quality attainable with the full baseline.
The configured 100-epoch training schedule matches the course baseline; the completed count is reported above.
Actual run settings
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"batch_size": 1,
"eval_num_frames": 32,
"num_workers": 2,
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"seed": 825
},
"model": {
"patch_size": 16,
"d_model": 256,
"n_layers": 4,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0004,
"weight_decay": 0.05,
"betas": [
0.9,
0.95
],
"warmup_steps": 100,
"grad_clip": 1.0,
"amp": true,
"vis_every": 1,
"output_dir": "outputs/lvsm"
},
"num_source_views": 2,
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/scene_splits.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 1,
"num_workers": 2,
"seed": 825
}
}
| Setting | PSNR (dB) | SSIM | Images |
|---|---|---|---|
| S=1,conditioning=correct | 14.32088 | 0.54649 | 196 |
| S=1,conditioning=shuffled | 11.96461 | 0.50584 | 196 |
| S=1,conditioning=none | 12.44046 | 0.54156 | 196 |
| S=2,conditioning=correct | 14.80931 | 0.54334 | 168 |
| S=2,conditioning=shuffled | 11.84142 | 0.49889 | 168 |
| S=2,conditioning=none | 12.36953 | 0.53706 | 168 |
| S=4,conditioning=correct | 15.08874 | 0.55381 | 112 |
| S=4,conditioning=shuffled | 12.05715 | 0.50690 | 112 |
| S=4,conditioning=none | 12.37061 | 0.54639 | 112 |
With 1 source views, correct-ray PSNR differs from shuffled rays by +2.356 dB and from removed rays by +1.880 dB.
With 2 source views, correct-ray PSNR differs from shuffled rays by +2.968 dB and from removed rays by +2.440 dB.
With 4 source views, correct-ray PSNR differs from shuffled rays by +3.032 dB and from removed rays by +2.718 dB.
Source-count comparisons also change which of the selected views serve as targets. They measure the supplied evaluation protocol, rather than a comparison with an identical target set. Ray-conditioning comparisons within one source count keep that target set fixed.









In the fixed-clip overfit example, the predictions preserve the desk, monitors, yellow chairs and the changes in viewpoint across targets. Fine edges retain a visible grain or patch texture, particularly on the floor and walls. This is a successful memorization check; it does not establish generalization to new rooms.
With correct rays, PSNR increases from 14.321 dB with one source view to 14.809 dB with two and 15.089 dB with four. SSIM is 0.54649, 0.54334 and 0.55381, respectively, so structural similarity is not monotonic. These source-count comparisons also change the target subset, as noted above; they do not isolate source count with identical targets.
Correct target rays outperform shuffled rays by 2.356–3.032 dB and zero rays by 1.880–2.718 dB across the three source counts. This indicates that the model uses target-camera conditioning. Shuffled rays are worse than zero rays in both PSNR and SSIM: an incorrect view query is more damaging here than an absent query. Zero-ray SSIM remains fairly close to correct-ray SSIM despite its lower PSNR, so SSIM alone understates the difference.
On the held-out dining-room example, the predictions reproduce broad beige and brown regions and a bright window area, but they do not resolve the table, chair legs, shelving or the recliner. Patch boundaries remain visible. Four source views improve the aggregate metrics without restoring those details. The gap from the fixed-clip overfit result shows that successful memorization does not imply strong generalization; reduced model capacity and optimization settings are possible contributors, but this run does not isolate their effects.
3. Multi-view geometry with Mini-VGGT
A frozen DINO encoder extracts image patches. Alternating attention first mixes tokens within each view and then across all views. The final frame and global features feed the camera and dense depth/confidence heads. Camera supervision uses L1 errors on translation, quaternion encoding and field of view; the depth objective combines confidence-weighted regression and spatial gradient terms.
3.1. Fixed-clip overfitting

| Fixed-clip metric | Initial | Final recorded epoch |
|---|---|---|
| Total loss | 5.41187 | 0.09692 |
| Camera loss | 0.34315 | 0.00077 |
| Depth loss | 3.69614 | 0.09305 |
| Depth AbsRel | 0.60586 | 0.06653 |
| Depth δ1 | 0.20003 | 0.91741 |
All metrics in this subsection replay the fixed training clip. The normal-run results below evaluate held-out scenes.
| Training setting | Actual value |
|---|---|
| Device / precision | Apple M3 Max (MPS) / fp32 |
| Image size | 224 × 224 |
| Views / sampling range | 8 / 32 frames |
| Batch size / seed | 1 / 825 |
| Learning rate / weight decay | 0.001 / 0.01 |
| Training scenes | 1 |
| Frozen encoder | dinov2_vits14 |
| Transformer width / frame-global pairs | 192 / 3 |
| Epochs completed / configured | 20 / 20 |
| Updates per fixed-clip epoch | 50 |
This run reduces the supplied model’s capacity: the baseline uses DINOv2 ViT-B/14, width 384 and eight frame/global attention pairs. The eight-view, 32-frame and 224-pixel geometry settings are retained. Recorded epochs describe the actual short training run, rather than the baseline’s 1,000-epoch schedule.
Last recorded epoch: 20. Recorded optimization epochs: 20.
Actual run settings
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 1,
"num_workers": 2
},
"model": {
"embed_dim": 192,
"num_heads": 6,
"num_layers": 3,
"predict_points": false,
"encoder_name": "dinov2_vits14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 20,
"learning_rate": 0.001,
"weight_decay": 0.01,
"output_dir": "outputs/vggt_overfit",
"overfit_scenes": 1,
"overfit_steps_per_epoch": 50,
"min_lr_ratio": 0.05,
"log_every": 10,
"warmup_epochs": 0,
"overfit_epochs": 20,
"overfit_learning_rate": 0.001
},
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/vggt_overfit_split.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 1,
"num_workers": 2,
"seed": 825
},
"run": {
"overfit": true,
"overfit_scenes": 1,
"steps_per_epoch": 50,
"max_val_batches": null,
"world_size": 1,
"precision": "fp32"
}
}


3.2. Normal training and held-out evaluation

| Training setting | Actual value |
|---|---|
| Device / precision | Apple M3 Max (MPS) / fp32 |
| Image size | 224 × 224 |
| Views / sampling range | 8 / 32 frames |
| Batch size / seed | 1 / 825 |
| Learning rate / weight decay | 0.0006 / 0.01 |
| Training scenes | 224 |
| Frozen encoder | dinov2_vits14 |
| Transformer width / frame-global pairs | 192 / 3 |
| Epochs completed / configured | 20 / 40 |
| Loss evaluation device | cpu |
| Checkpoint-safe epoch cap | 20 |
This run reduces the supplied model’s capacity: the baseline uses DINOv2 ViT-B/14, width 384 and eight frame/global attention pairs. The eight-view, 32-frame and 224-pixel geometry settings are retained. Recorded epochs describe the actual short training run, rather than the baseline’s 1,000-epoch schedule.
Last recorded epoch: 20. Recorded optimization epochs: 20.
Validation selects epoch 18 by minimum combined camera/depth loss. Recorded epoch times include the full validation pass and exclude checkpoint writes and curve plotting; MPS was shared with Mini-LVSM.
Training resumed from the saved epoch-7 checkpoint with its optimizer state and original 40-epoch learning-rate schedule. The differentiable loss calculation moved from MPS to CPU to avoid variable-shape loss overhead; predictions still come from the MPS model. The objective and its weights are unchanged.
The interrupted, unsaved portion of epoch 8 was discarded. Its final logged step was 40; the number of additional unlogged updates is unknown. The supplied resume loader resets its sampling seed and data order, so this run does not preserve the uninterrupted loader RNG trajectory.
Actual run settings
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 1,
"num_workers": 2
},
"model": {
"embed_dim": 192,
"num_heads": 6,
"num_layers": 3,
"predict_points": false,
"encoder_name": "dinov2_vits14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 40,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"output_dir": "outputs/vggt_train",
"overfit_scenes": 1,
"overfit_steps_per_epoch": 50,
"min_lr_ratio": 0.05,
"log_every": 10,
"warmup_epochs": 0,
"overfit_epochs": 20,
"overfit_learning_rate": 0.001
},
"data": {
"root": "data/scannetpp/processed",
"eval_root": "data/scannetpp/prepared_eval",
"eval_num_frames": 32,
"split_file": "configs/scannetpp/iphone/scene_splits.json",
"num_views": 8,
"sampling_range": 32,
"image_size": 224,
"batch_size": 1,
"num_workers": 2,
"seed": 825
},
"run": {
"overfit": false,
"overfit_scenes": 1,
"steps_per_epoch": null,
"max_val_batches": null,
"world_size": 1,
"precision": "fp32",
"stop_after_epoch": 20,
"loss_device": "cpu"
}
}



| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.18410 | 0.21100 |
| Depth δ1 | 0.72269 | 0.66019 |
| Rotation AUC @5° | 0.09745 | 0.11709 |
| Rotation AUC @15° | 0.24949 | 0.29277 |
| Rotation AUC @30° | 0.42266 | 0.47997 |
| Translation AUC @5° | 0.00459 | 0.00077 |
| Translation AUC @15° | 0.02764 | 0.02551 |
| Translation AUC @30° | 0.08431 | 0.09252 |
| Joint pose AUC @5° | 0.00077 | 0.00026 |
| Joint pose AUC @15° | 0.01412 | 0.01497 |
| Joint pose AUC @30° | 0.06841 | 0.07058 |
| Split | Scenes | All camera pairs | Pairs with undefined translation (subset) |
|---|---|---|---|
| Validation | 28 | 784 | 0 |
| Test | 28 | 784 | 0 |
Depth metrics include all valid ground-truth pixels. Rotation AUC uses all camera pairs. Translation and joint-pose AUC omit pairs whose ground-truth translation direction is undefined and allow translation sign ambiguity.
On the fixed training clip, the predicted cloud recovers the room layout, floor and camera trajectory. The predicted surfaces are somewhat thicker and less crisp than the measured depth cloud. Both panels use the same coordinates, axis limits, RGB colors and GT-valid pixels; no alignment is fitted to the prediction.
The four held-out scenes show substantially larger errors than the memorized clip. In 09c1414f1b, the predicted cloud retains broad room regions but thickens and mixes surfaces. In 21d970d8de, it compresses the extent of the reconstructed structure. In 280b83fcf3, the tall ground-truth structure becomes a shorter, denser cloud and its separated surfaces are less distinct. In 2b1dc6d6a5, planar surfaces become diffuse, with scattered points beyond the ground-truth structure.
Camera prediction is a major limitation: predicted camera centers cluster near the reference origin, while the ground-truth trajectories span larger baselines, particularly in the fourth scene. Errors in both camera poses and per-view depth therefore accumulate when the views are fused into a point cloud. The comparison uses the same canonical coordinates, axis limits, sampled pixels and RGB colors for GT and prediction, with no fitted alignment. Recognizable colors do not by themselves establish accurate geometry.
Validation depth AbsRel is 0.18410 with δ1 = 0.72269; test AbsRel is 0.21100 with δ1 = 0.66019. Test depth accuracy is lower on both measures. Joint pose AUC at 30° is only 0.06841 on validation and 0.07058 on test, consistent with the inaccurate camera trajectories in the comparisons. The higher test rotation AUC does not remove the translation-direction errors.
The normal run covers all 224 training scenes but stops after 20 epochs of a 40-epoch schedule and uses a smaller encoder and transformer than the supplied baseline. Its weak held-out pose accuracy contrasts with the successful overfit check. Longer training or greater capacity may improve generalization, but neither explanation is isolated by these experiments.
Artifact status
All required artifact categories are present. Measurements and settings are reported below.
Reproduction
Setup and experiment commands · Original assignment overview
python build_report.py python verify_artifacts.py python package_submission.py
Starter code and task definitions: learning3d/assignment4_f26. Training and evaluation use the course ScanNet++ iPhone split. This report displays saved outputs from the implementation; the supplied TA reference is not a submission result.
Recorded environment · Package versions
Implementation checks
23 passed in 4.08s. Geometry checks cover resize and camera conventions, canonicalization and reprojection. Model checks cover patch/token construction, output shapes, gradients and loss behavior.
python -m pytest -q tests