Multi-View Data Preprocessing
1.2–1.3 Unprojection and Canonicalization
Scene 1: 036bce3393
Scene 2: 076c822ecc
Scene 3: 079a326597
Scene 4: 07f5b601ee
Scene 5: 07ff1c45bb
Canonicalization: Camera poses and 3D points are expressed in the first sampled camera’s coordinate frame. Camera translations, depth, and points are divided by the valid-point centroid’s distance from the origin. The first camera has identity pose, and the centroid distance becomes approximately 1 without subtracting the centroid.
World-space ray directions: For pixel (u, v), the direction is computed as R K−1[u, v, 1]T and normalized to unit length. The rotation maps the direction into world coordinates; translation determines the ray origin and does not affect its direction.
Plücker moment: A ray is represented by its unit direction d and moment m = o × d, where o is the ray origin. The moment encodes the line’s position relative to the coordinate origin and remains unchanged when o moves along the ray.
1.7 Final Data Check
The maximum point reprojection error is 0.000123020 pixels; the maximum ray reprojection error is 0.000111086 pixels. Both are below 0.01 pixels, and all four reported centroid distances are 1.0.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.777836385183036e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011736395390471444,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}
Novel-View Synthesis with Mini-LVSM
2.1 Fixed-Batch Overfitting
Final overfit PSNR: 30.8939 dB, measured on the same fixed batch after training.
2.2 Training and Evaluation
Training settings: ScanNet++ training split, 8 views per sample, 2 source views and 6 target views, sampling range 32, image size 224 × 224, batch size 4, and seed 825. The model uses 8 × 8 patches, transformer width 384, 8 blocks, attention head dimension 64, MLP ratio 4, and query/key normalization. I use MSE loss and AdamW with learning rate 0.0002, weight decay 0.05, betas (0.9, 0.95), gradient clipping at 1.0, and BF16 autocast. The saved log contains 1,000 epochs (indices 0–999). The best checkpoint is epoch 956 (zero-based), selected by validation PSNR.
Every RGB comparison uses columns in the order: source views, ground truth, prediction. Evaluation uses best.pt on all 28 validation scenes, with 8 views selected from the first 32 prepared frames, 224 × 224 images, and seed 825. correct supplies the true target rays; shuffled supplies another target view’s rays; none supplies zero target rays.
1 Source View
Correct rays: PSNR = 16.2029 dB; SSIM = 0.6014.
Shuffled rays: PSNR = 11.5484 dB; SSIM = 0.4754.
None rays: PSNR = 12.3218 dB; SSIM = 0.5246.
2 Source Views
Correct rays: PSNR = 17.7453 dB; SSIM = 0.6203.
Shuffled rays: PSNR = 11.1273 dB; SSIM = 0.4519.
None rays: PSNR = 12.3591 dB; SSIM = 0.5217.
4 Source Views
Correct rays: PSNR = 19.0686 dB; SSIM = 0.6468.
Shuffled rays: PSNR = 11.1999 dB; SSIM = 0.4477.
None rays: PSNR = 12.3778 dB; SSIM = 0.5304.
Effect of source-view count: With correct rays, PSNR increases from 16.2029 to 17.7453 and 19.0686 dB as the number of sources increases from 1 to 2 and 4; SSIM increases from 0.6014 to 0.6203 and 0.6468. These results are consistent with additional source views providing more scene information. The improvement from 2 to 4 sources shows that additional views remain useful beyond the 2-source training setting. Complementary viewpoints can reveal occluded surfaces and provide additional observations of scene appearance, reducing ambiguity in target-view synthesis. Both pixel accuracy and structural similarity improve across the evaluated source counts.
Effect of ray conditioning: At 2 source views, shuffled rays reduce PSNR from 17.7453 to 11.1273 dB and SSIM from 0.6203 to 0.4519; zero rays give 12.3591 dB and 0.5217. Increasing the source-view count does not recover the correct-ray performance when the target rays are wrong or missing. This indicates that the model uses the target-ray input to select the requested viewpoint.
2.3 Bonus: Loss Function Exploration
Question and hypothesis: Does adding an L1 image reconstruction term to the baseline MSE loss improve novel-view synthesis? The hypothesis is that MSE + L1 improves PSNR, SSIM, and visual sharpness compared with MSE alone.
Loss definition: The baseline minimizes L = mean((prediction − GT)²). The alternative minimizes L = L_MSE + 0.1 × L_L1, where L_L1 = mean(|prediction − GT|). Both terms use RGB values in [0, 1] and average over all target views, color channels, and pixels, without a depth-validity mask.
Experimental settings: Both runs use the same Mini-LVSM architecture and data settings as Q2.2: 8 × 8 patches, width 384, 8 transformer blocks, 8 views per sample (2 sources and 6 targets), sampling range 32, 224 × 224 images, batch size 4, and seed 825. Both logs contain 1,000 training epochs. Optimization uses AdamW with learning rate 0.0002, weight decay 0.05, betas (0.9, 0.95), 200 warmup steps followed by cosine decay, gradient clipping at 1.0, and BF16 autocast with float32 loss computation. Each run's best checkpoint is selected by validation PSNR: epoch 956 for MSE and epoch 971 for MSE + L1 (zero-based). Evaluation uses the same 28 validation scenes, 8 views from the first 32 prepared frames, 2 source views, correct target rays, image size 224, and seed 825, totaling 168 target images per model.
Quantitative Results
MSE baseline:
PSNR: 17.7453 dB
SSIM: 0.620325
MSE: 0.0234631
MSE + 0.1 × L1:
PSNR: 17.7533 dB
SSIM: 0.625468
MSE: approximately 0.02334
L1: approximately 0.1031
Total loss: 0.0336462
Metric changes: PSNR increases by 0.0081 dB and SSIM increases by 0.005143. The reported MSE loss component also decreases slightly.
Training behavior: The PSNR curves approach nearly the same final level, while the MSE + L1 SSIM curve remains above the baseline through much of the later training. Adding an L1 loss term can accelerate convergence and improve training stability to some extent, while slightly improving the final model performance.
Visual Comparison
Visual sharpness: Both models recover the broad room layout, but the predictions remain visibly blurred around chair legs, window frames, and shelf details. The L1 variant changes some local contours and contrast, but these examples do not show a clear improvement in sharpness.
Possible explanation: MSE gives large pixel residuals a quadratic penalty, whereas L1 gives residuals a linear penalty and changes their relative contribution during optimization. The additional term may help preserve some local appearance and structure, consistent with the SSIM increase. However, L1 does not explicitly supervise image gradients or perceptual features, so it does not guarantee sharper edges. Ambiguity in unseen target content can still produce smooth predictions under either pixel reconstruction objective.
Conclusion and limitations: The hypothesis is only partially supported: PSNR improves by a very small amount and SSIM improves modestly, while a consistent sharpness gain is not established by the saved visualizations. Adding L1 provides pixel-level supervision but does not explicitly supervise perceptual features, which may limit improvements in visual detail. A perceptual loss is therefore a promising additional objective to investigate. Model capacity and the training budget may also limit performance; increasing the parameter count and training for more epochs may also improve reconstruction quality.
Multi-View Geometry with Mini-VGGT
3.1 Single-Scene Overfitting
3.2 Training and Evaluation
Training settings: 224 training scenes, 8 views per sample, sampling range 32, 224 × 224 images, batch size 16, and seed 825. The encoder is a frozen DINOv2 ViT-B/14. The geometry transformer uses width 384, 6 attention heads, and 8 frame/global pairs (16 blocks), followed by a camera head with 4 cross-view blocks and a depth head with RGB refinement. I train for 1,000 epochs using AdamW, learning rate 0.0006, weight decay 0.01, 10 warmup epochs, cosine decay with minimum learning-rate ratio 0.05, gradient clipping at 1.0, and BF16 on one GPU. The objective is 5 × camera loss + depth loss. The best checkpoint is selected by validation loss.
Validation and Test Metrics
Both evaluations use best.pt, 8 views from the first 32 prepared frames, image size 224, and seed 825. Each complete split contains 28 scenes and 784 camera pairs. Depth metrics use all GT-valid pixels. AUC values below are in [0, 1].
Validation
Depth AbsRel ↓: 0.158346
Depth δ1 ↑: 0.801337
Rotation AUC @ 5° ↑: 0.217602
Rotation AUC @ 15° ↑: 0.554422
Rotation AUC @ 30° ↑: 0.733248
Translation AUC @ 5° ↑: 0.021429
Translation AUC @ 15° ↑: 0.095663
Translation AUC @ 30° ↑: 0.220918
Joint Pose AUC @ 5° ↑: 0.004592
Joint Pose AUC @ 15° ↑: 0.070493
Joint Pose AUC @ 30° ↑: 0.197874
Test
Depth AbsRel ↓: 0.173307
Depth δ1 ↑: 0.759338
Rotation AUC @ 5° ↑: 0.190051
Rotation AUC @ 15° ↑: 0.515136
Rotation AUC @ 30° ↑: 0.712117
Translation AUC @ 5° ↑: 0.014796
Translation AUC @ 15° ↑: 0.078146
Translation AUC @ 30° ↑: 0.197236
Joint Pose AUC @ 5° ↑: 0.005357
Joint Pose AUC @ 15° ↑: 0.060119
Joint Pose AUC @ 30° ↑: 0.180527
GT and Predicted Geometry: Four Validation Scenes
Scene 1: 09c1414f1b
Scene 2: 21d970d8de
Scene 3: 280b83fcf3
Scene 4: 2b1dc6d6a5
Held-out errors: Depth AbsRel increases from 0.158346 on validation to 0.173307 on test, while δ1 decreases from 0.801337 to 0.759338. Rotation AUC @ 30° is 0.733248 / 0.712117 (validation / test), but translation AUC @ 30° is only 0.220918 / 0.197236. The lower joint pose AUC (0.197874 / 0.180527 at 30°) indicates that accurate relative translation remains a limitation. Depth and pose errors both affect the point clouds reconstructed from predictions, so a plausible depth map alone does not guarantee consistent multi-view geometry.
3.3 Bonus: Point-Map Prediction
Research question: Does directly predicting a world-space point map improve reconstruction accuracy compared with reconstructing points from predicted cameras and depth? And whether adding point supervision also improves the camera-and-depth reconstruction pathway within the same model?
Point prediction head: A linear layer maps each aggregated patch feature to 3 × p² values, which are rearranged into a dense XYZ map. A residual RGB refinement network concatenates this map with the input RGB and applies a 3 × 3 convolution from 6 to 32 channels, GELU, and a 3 × 3 convolution from 32 to 3 channels. The final convolution is initialized to zero. The output has shape (B, V, 3, H, W), uses float32, and has no positivity activation because XYZ coordinates can be negative. Points are predicted in the shared canonical first-camera frame.
Point supervision: The point loss is the mean absolute XYZ coordinate error over GT-valid pixels, averaged across coordinates and all valid pixels. The mask requires valid, positive, finite GT depth and finite GT points. The baseline objective is 5 × L_camera + L_depth; the point model uses 5 × L_camera + L_depth + 1 × L_point. Camera and depth heads remain active, so the new model supports both direct point prediction and reconstruction by unprojection.
Experimental settings: Both runs use 224 training scenes, 8 views per sample, sampling range 32, 224 × 224 images, batch size 16, and seed 825. Both retain the frozen DINOv2 ViT-B/14 encoder and the same geometry transformer (width 384, 6 heads, 8 frame/global attention pairs), camera head, and depth head. Each run trains for 1,000 epochs on one GPU using BF16, AdamW with learning rate 0.0006 and weight decay 0.01, 10 warmup epochs, cosine decay with minimum learning-rate ratio 0.05, and gradient clipping at 1.0. The point head and its loss are the experimental additions. Each run selects its best checkpoint using its own total validation loss.
Evaluation protocol: Both best checkpoints are evaluated on the complete validation and test splits, with 28 scenes per split, 8 views from the first 32 prepared frames, image size 224, and seed 825. Validation contains 10,347,720 GT-valid pixels and test contains 10,517,881; each split has 784 camera pairs. Point error is the valid-pixel-weighted mean Euclidean distance between predicted and GT XYZ at corresponding pixels. All three reconstruction pathways use the same GT depth-validity mask and canonical coordinates. Meter-space error multiplies each pixel's normalized distance by its scene's GT scene scale before aggregation.
Point Reconstruction Results
Validation — lower is better
Baseline, camera + depth unprojection:
Normalized point L2: 0.268391
Point L2 in meters: 0.542356
Point model, direct prediction:
Normalized point L2: 0.252470
Point L2 in meters: 0.514669
Point model, camera + depth unprojection:
Normalized point L2: 0.247324
Point L2 in meters: 0.503423
Test — lower is better
Baseline, camera + depth unprojection:
Normalized point L2: 0.257238
Point L2 in meters: 0.510092
Point model, direct prediction:
Normalized point L2: 0.234176
Point L2 in meters: 0.441911
Point model, camera + depth unprojection:
Normalized point L2: 0.223313
Point L2 in meters: 0.415870
Comparison of the two pathways: Relative to baseline unprojection, direct point prediction reduces normalized point error by 5.93% on validation and 8.97% on test. However, unprojection within the point-supervised model performs best, reducing error by 7.85% and 13.19% relative to the baseline. Direct prediction has 2.08% higher normalized error than this model's own unprojection on validation and 4.86% higher error on test. Thus, the added point-supervised model improves reconstruction, but its direct prediction head does not outperform its camera-and-depth pathway.
Depth and Camera Results
The values below compare the baseline with the point-supervised model. AUC entries list thresholds in the order 5° / 15° / 30° and range from 0 to 1. Depth errors are lower-is-better; δ1 and AUC are higher-is-better.
Validation — baseline
Depth AbsRel: 0.158346
Depth δ1: 0.801337
Normalized depth RMSE: 0.204834
Rotation AUC: 0.217602 / 0.554422 / 0.733248
Translation AUC: 0.021429 / 0.095663 / 0.220918
Joint Pose AUC: 0.004592 / 0.070493 / 0.197874
Validation — point model
Depth AbsRel: 0.128646
Depth δ1: 0.844111
Normalized depth RMSE: 0.175149
Rotation AUC: 0.227551 / 0.555187 / 0.728656
Translation AUC: 0.023214 / 0.104507 / 0.220153
Joint Pose AUC: 0.011735 / 0.084524 / 0.195918
Test — baseline
Depth AbsRel: 0.173307
Depth δ1: 0.759338
Normalized depth RMSE: 0.215537
Rotation AUC: 0.190051 / 0.515136 / 0.712117
Translation AUC: 0.014796 / 0.078146 / 0.197236
Joint Pose AUC: 0.005357 / 0.060119 / 0.180527
Test — point model
Depth AbsRel: 0.140790
Depth δ1: 0.829336
Normalized depth RMSE: 0.197131
Rotation AUC: 0.233673 / 0.551786 / 0.727381
Translation AUC: 0.018367 / 0.082143 / 0.203954
Joint Pose AUC: 0.007908 / 0.063946 / 0.184269
Geometry Comparisons: Four Validation Scenes
Scene 1: 09c1414f1b
Baseline and point-supervised model
Point-supervised model: two reconstruction pathways
Scene 2: 21d970d8de
Baseline and point-supervised model
Point-supervised model: two reconstruction pathways
Scene 3: 280b83fcf3
Baseline and point-supervised model
Point-supervised model: two reconstruction pathways
Scene 4: 2b1dc6d6a5
Baseline and point-supervised model
Point-supervised model: two reconstruction pathways
Effect on the original tasks: Depth improves on both splits: AbsRel decreases from 0.158346 to 0.128646 on validation and from 0.173307 to 0.140790 on test, while δ1 increases from 0.801337 to 0.844111 and from 0.759338 to 0.829336. Camera gains are less uniform. All reported test AUC values improve, but validation rotation, translation, and joint AUC at 30° decrease slightly. Adding point supervision improves depth prediction and reconstruction, and also yields small gains in several camera estimation metrics, particularly on the test split.
Possible explanations: The point objective supplies an additional geometric learning signal to the shared transformer, which may improve the features used by the depth and camera heads. Unprojection constrains each point to the ray defined by the predicted camera and pixel, whereas the direct head predicts three coordinates without an explicit constraint tying them to its predicted camera and depth. This may answer why unprojection remains more accurate within the point model.
Conclusion and limitations: Adding a direct point head and point loss improves point reconstruction and depth metrics relative to the baseline, but the most accurate points come from unprojecting the point-supervised model's cameras and depth. These results favor point supervision as an auxiliary task over replacing unprojection with this direct head.