Parth Burujwale Andrew ID: pburujwa 16-825: Learning for 3D Vision
This assignment explores multi-view geometry preprocessing, novel-view synthesis with Mini-LVSM, and camera/depth reconstruction with Mini-VGGT using ScanNet++ iPhone images.
I updated the cropped camera intrinsics to match the resized image dimensions. Focal lengths scale independently along the horizontal and vertical axes.
The dataloader already handles the center-crop adjustment.
For each pixel, I computed its camera-space point using:
\[ \mathbf{X}_{c} = zK^{-1}[u,v,1]^T. \]
The camera-to-world pose then transforms the point:
\[ \mathbf{X}_{w} = R\mathbf{X}_{c} + \mathbf{t}. \]
The supplied depth is optical-axis Z-depth, so the backprojected camera vectors are not normalized before multiplication by depth. The provided pipeline masks invalid output points.
Unprojected RGB-D points and camera poses in the original world coordinate system, measured in meters.
I expressed every camera pose in the first sampled camera’s coordinate frame:
\[ T_{\mathrm{reference}\leftarrow i} = T_{\mathrm{world}\leftarrow 0}^{-1} T_{\mathrm{world}\leftarrow i}. \]
I then used the provided scene-scale calculation:
\[ s = \left\| \operatorname{mean}_{\mathrm{valid}} (\mathbf{X}_{\mathrm{reference}}) \right\|_2. \]
Camera 0 remains at the origin, and the valid-point centroid has distance approximately one from it.
Geometry in the first camera’s frame with shared scale normalization. The submission screenshot shows the first camera center and valid-point centroid distance.
Each ray originates at the camera center. Its direction is obtained by backprojecting the pixel through the intrinsics, rotating into the canonical world frame, and normalizing:
\[ \mathbf{o} = \mathbf{t}, \qquad \mathbf{d} = \frac{RK^{-1}[u,v,1]^T} {\left\|RK^{-1}[u,v,1]^T\right\|_2}. \]
Translation determines the origin, while rotation determines the world-space direction.
I represented each ray using six channels:
\[ [\mathbf{d},\mathbf{m}], \qquad \mathbf{m} = \mathbf{o}\times\mathbf{d}. \]
The direction specifies the ray orientation. The moment encodes the line’s location relative to the origin and remains unchanged when the origin moves along the same line.
The dataset contains 224 training, 28 validation and 28 test scenes.
The training loader randomly samples eight views from a sampling window and applies the implemented preprocessing functions. The first sampled view defines the reference camera.
Validation and test use prepared geometry and fixed view selection. RGB remains valid independently of the depth-validity mask.
All four checked training samples satisfied the required reprojection threshold of 0.01 pixels.
| Sample | Valid depth fraction | Maximum point error (px) | Maximum ray error (px) | Centroid distance |
|---|---|---|---|---|
e050c15a8d_frame_007080 |
0.977913 | 0.00010845 | 0.00009414 | 1.0 |
5f99900f09_frame_002820 |
0.990312 | 0.00012302 | 0.00010790 | 1.0 |
47b37eb6f9_frame_002720 |
0.960995 | 0.00009537 | 0.00008217 | 1.0 |
4c5c60fa76_frame_006690 |
0.751672 | 0.00010784 | 0.00009537 | 1.0 |
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010844774078577757,
"max_ray_reprojection_error_px": 9.41436446737498e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.0001078959321603179,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.5367431640625e-05,
"max_ray_reprojection_error_px": 8.217109279939905e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010783717152662575,
"max_ray_reprojection_error_px": 9.53696362557821e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [2, 8, 3, 224, 224],
"depth": [2, 8, 1, 224, 224],
"depth_valid": [2, 8, 1, 224, 224],
"K": [2, 8, 3, 3],
"frame_numbers": [2, 8],
"T_world_from_camera": [2, 8, 4, 4],
"original_world_from_reference": [2, 4, 4],
"scene_scale": [2],
"points_world": [2, 8, 3, 224, 224],
"ray_origins": [2, 8, 3, 224, 224],
"ray_directions": [2, 8, 3, 224, 224],
"plucker": [2, 8, 6, 224, 224]
}
}I trained on one fixed batch for 2,000 optimization steps.
| Final fixed-batch metric | Result |
|---|---|
| MSE | 0.00086 |
| PSNR | 30.95 dB |
| SSIM | 0.8706 |
| Runtime | 240 seconds |
The final PSNR exceeded the approximately 30 dB sanity-check target. This demonstrates that the model can memorize the fixed batch.
Columns: two source views, target ground truth, and prediction.
| Setting | Value |
|---|---|
| Training scenes | 224 |
| Image resolution | 224 × 224 |
| Sampled views | 8 |
| Source / target views | 2 / 6 |
| Batch size | 4 |
| Epochs | 100 |
| Patch size | 8 |
| Transformer width | 384 |
| Transformer blocks | 8 |
| Parameters | Approximately 14.6 million |
| Optimizer | AdamW |
| Learning rate | 0.0002 |
| Weight decay | 0.05 |
| Adam betas | 0.9, 0.95 |
| Warmup | 200 steps |
| Gradient clipping | 1.0 |
| Precision | bfloat16 autocast |
| Hardware | NVIDIA H100 80 GB |
| Seed | 825 |
Training and validation MSE and PSNR, together with validation SSIM.
Training and validation improved before approaching a plateau. The best validation PSNR was 16.1409 dB at epoch 91, with SSIM 0.5601.
Normal-training visualization showing source images, target ground truth and predicted RGB.
I evaluated best.pt with 1, 2 and 4 source views under
three target-ray conditions:
| Sources | Target rays | PSNR ↑ | SSIM ↑ | MSE ↓ | Target images |
|---|---|---|---|---|---|
| 1 | Correct | 15.19 | 0.5571 | 0.03523 | 196 |
| 1 | Shuffled | 11.79 | 0.4862 | 0.07307 | 196 |
| 1 | None | 12.46 | 0.5297 | 0.05921 | 196 |
| 2 | Correct | 16.14 | 0.5601 | 0.02927 | 168 |
| 2 | Shuffled | 11.47 | 0.4719 | 0.07824 | 168 |
| 2 | None | 12.42 | 0.5264 | 0.05960 | 168 |
| 4 | Correct | 16.87 | 0.5757 | 0.02438 | 112 |
| 4 | Shuffled | 11.58 | 0.4746 | 0.07848 | 112 |
| 4 | None | 12.45 | 0.5357 | 0.05905 | 112 |
Each row contains the source views followed by target ground truth and prediction.
Correct-ray PSNR increased from 15.19 dB with one source to 16.87 dB with four sources. Additional source observations can provide more scene information. However, the evaluation target sets also change with source count, so these results are not a comparison on identical target images.
With two sources, shuffling rays reduced PSNR by 4.67 dB, while removing rays reduced it by 3.72 dB. Correct target-camera information is therefore important to the model’s predictions.
Shuffled rays produced layouts inconsistent with the target. With zero target rays, predictions were nearly identical across target views because the model no longer received distinct target-camera queries.
Even with correct rays, held-out predictions remained blurry and lost furniture details. The strong fixed-batch overfit result did not translate into equally detailed reconstruction of unseen scenes.
I trained on one fixed eight-view training clip to verify that the model could fit camera poses and depth.
| Final fixed-clip metric | Result |
|---|---|
| Depth AbsRel ↓ | 0.0371 |
| Depth δ1 ↑ | 0.9549 |
| Mean rotation error ↓ | 0.1298° |
| Mean translation-direction error ↓ | 0.2404° |
| Joint pose AUC @ 5° ↑ | 0.9929 |
Total, camera and depth losses decrease substantially during single-clip fitting.
The depth objective can become negative because it includes the confidence term \(-0.2\log(c)\). A negative total loss is therefore possible.
| Ground truth | Predicted camera + depth |
|---|---|
![]() |
![]() |
The predicted camera arrangement and broad room structure closely matched the fixed clip’s ground truth. Predicted depth was smoother and retained less fine detail.
| Setting | Value |
|---|---|
| Training scenes | 224 |
| Input | RGB only |
| Image resolution | 224 × 224 |
| Views per sample | 8 |
| Sampling range | 32 |
| Batch size | 16 |
| Configured training epochs | 1,000 |
| Encoder | Frozen DINOv2 ViT-B/14 |
| Aggregation | 8 frame/global pairs |
| Transformer width | 384 |
| Attention heads | 6 |
| Camera-head transformer blocks | 4 |
| Learning rate | 0.0006 |
| Weight decay | 0.01 |
| Warmup | 10 epochs |
| Camera / depth loss weights | 5.0 / 1.0 |
| Hardware | NVIDIA H100 80 GB |
| Best-checkpoint selection | Lowest validation loss |
Training and validation total, camera and depth losses.
Both training and validation losses decreased, with smaller improvements late in training. Validation loss did not show a sustained upward trend.
I evaluated the checkpoint selected by validation loss on the complete validation and test splits.
Both evaluations used 8 views from the first 32 prepared frames, 224 × 224 images and seed 825, without a maximum-batch restriction.
| Metric | Validation | Test |
|---|---|---|
| Scenes | 28 | 28 |
| Depth AbsRel ↓ | 0.1479 | 0.1493 |
| Depth δ1 ↑ | 0.8161 | 0.8248 |
| Rotation AUC @ 5° ↑ | 0.1972 | 0.2089 |
| Rotation AUC @ 15° ↑ | 0.5293 | 0.5475 |
| Rotation AUC @ 30° ↑ | 0.7126 | 0.7298 |
| Translation AUC @ 5° ↑ | 0.0179 | 0.0122 |
| Translation AUC @ 15° ↑ | 0.0834 | 0.0772 |
| Translation AUC @ 30° ↑ | 0.1924 | 0.1819 |
| Joint pose AUC @ 5° ↑ | 0.0046 | 0.0036 |
| Joint pose AUC @ 15° ↑ | 0.0579 | 0.0576 |
| Joint pose AUC @ 30° ↑ | 0.1662 | 0.1623 |
AUC values are reported on a 0–1 scale.
09c1414f1b| Ground truth | Predicted camera + depth |
|---|---|
![]() |
![]() |
The prediction retained broad scene structure but blurred furniture boundaries. The predicted cameras were more tightly clustered than in ground truth.
21d970d8de| Ground truth | Predicted camera + depth |
|---|---|
![]() |
![]() |
The broad room layout remained recognizable, while surfaces were smoother and some structural boundaries were distorted.
280b83fcf3| Ground truth | Predicted camera + depth |
|---|---|
![]() |
![]() |
The predicted hallway appeared compressed and curved relative to ground truth, with reduced doorway detail.
2b1dc6d6a5| Ground truth | Predicted camera + depth |
|---|---|
![]() |
![]() |
The prediction preserved the rough layout but showed smeared chair/desk geometry and overlapping surfaces.
Depth performance was similar on validation and test, with AbsRel near 0.15 and δ1 above 0.81 on both splits.
Translation-direction estimation remained substantially weaker than rotation estimation, limiting joint pose AUC. The qualitative examples also showed camera-arrangement differences and imperfect alignment of points reconstructed across views.
Compared with single-clip overfitting, held-out reconstructions lost fine detail and exhibited greater distortion. The model recovered coarse scene structure, but depth smoothing and camera-pose errors limited detailed multi-view consistency.