Q1. Multi-View Data Preprocessing (20 points)
1. Depth Unprojection
We unprojected the depth maps into 3D point clouds using the camera intrinsics and transformed the points into world coordinates using the camera poses. The visualization below shows the reconstructed scene from eight sampled views in the original world coordinate system.
2. Camera Canonicalization
We use the first camera as the reference frame and scale the camera positions, depth, and 3D points so the valid-point centroid is 1 unit from the origin.
3. World-Space Ray Directions
We use the camera intrinsics to convert each pixel into a ray direction, rotate it into world coordinates, and normalize it to unit length.
4. Plücker Moment
The moment is computed as
m = o × d
and describes the line's position relative to the origin.
We combine it with the direction to form six channels,
[d, m].
5. Data Sanity Check
We checked four training samples using
check_data.py.
The tensor shapes matched the specification, and
valid depth ratios ranged from 75.2% to 99.0%.
The maximum point and ray reprojection errors were
approximately 0.000123 and 0.000111 pixels,
respectively, both below the required 0.01 pixels.
The centroid distance was 1.0 for all samples,
as expected.
| Metric | Result |
|---|---|
| Samples checked | 4 |
| Tensor shapes | Passed |
| Valid depth ratio | 75.2% - 99.0% |
| Max point reprojection error | 0.000123 px |
| Max ray reprojection error | 0.000111 px |
| Centroid distance | 1.000 |
Q2. Novel-View Synthesis with Mini-LVSM (40 points)
1. Overfitting
We trained the Mini-LVSM model on a fixed batch for 2,000 steps to verify that the model could overfit a small set of training examples. The model contains 14.6 million parameters and uses two source views.
After 2,000 steps, the model achieved a validation PSNR of 30.92 dB and an SSIM of 0.8703, meeting the target PSNR of approximately 30 dB. The final training loss was 0.00684, and the validation loss was 0.00087. Training took approximately 2.11 hours on the GPU.
| Metric | Result |
|---|---|
| Training Steps | 2,000 |
| Source Views | 2 |
| Train PSNR | 25.40 dB |
| Validation PSNR | 30.92 dB |
| Validation SSIM | 0.8703 |
| Training Time | 7,580 s (2.11 hours) |
Overall, the overfitting experiment confirms that the Mini-LVSM implementation can learn the mapping from source views and target rays to the target RGB image.
2. Full Training and Evaluation
Training Results
We trained Mini-LVSM for 100 epochs with 2 source views, batch size 4, and learning rate 2e-4. Training took about 6.2 hours.
The training loss decreased from 0.06344 to 0.03394. The best validation PSNR was 16.13 dB at epoch 97, and the best SSIM was 0.5600 at epoch 81.
| Metric | Result |
|---|---|
| Epochs | 100 |
| Batch Size | 4 |
| Source Views | 2 |
| Learning Rate | 2e-4 |
| Best Validation PSNR | 16.13 dB |
| Best Validation SSIM | 0.5600 |
| Training Time | 6.2 hours |
Training Curves
Both training and validation losses decreased, while PSNR and SSIM improved over time. The curves became more stable toward the end, showing that the model was converging.
Training Visualizations
The predictions improved as training progressed. The model learned the main scene structure, but some details remained blurry.
Evaluation and Ablation Study
We evaluated the model using 1, 2, and 4 source views with correct, shuffled, and no ray conditioning. The table shows PSNR (dB) / SSIM.
| Source Views | Correct | Shuffled | None |
|---|---|---|---|
| 1 | 15.23 / 0.5568 | 11.77 / 0.4853 | 12.48 / 0.5267 |
| 2 | 16.13 / 0.5591 | 11.46 / 0.4718 | 12.46 / 0.5262 |
| 4 | 16.89 / 0.5748 | 11.57 / 0.4732 | 12.48 / 0.5362 |
The evaluation used 196, 168, and 112 images for 1, 2, and 4 source views, respectively.
Qualitative Evaluation
One Source View
Two Source Views
Four Source Views
Discussion
With correct ray conditioning, using more source views improved PSNR from 15.23 dB to 16.89 dB. The predictions also preserved more scene details.
Shuffled or missing rays significantly reduced reconstruction quality, even with more source views. This shows that correct camera ray information is important for novel-view synthesis.
Q3. Multi-View Geometry with Mini-VGGT (40 points)
1. Overfitting
We trained Mini-VGGT on one fixed training clip for 40 epochs, with 25 steps per epoch (1,000 steps in total).
Loss Curves
The total, camera, and depth losses all decreased substantially and became more stable toward the end of training.
Ground Truth and Predicted Geometry
The predicted camera arrangement and main room structure are close to the ground truth. Some noise and smoother depth boundaries remain in the prediction. The first predicted camera center is approximately (0.0000, 0.0001, -0.0001), close to the origin. The predicted valid-point centroid distance is 0.997, compared with 1.000 for the ground truth.
2. Full Training and Evaluation
Training Settings
We trained Mini-VGGT on all 224 training scenes for 1,000 epochs, using batch size 8 and 28 steps per epoch. Each sample contains 8 views sampled within a range of 32 frames at 224 × 224 resolution. We used a frozen DINOv2 ViT-B/14 encoder, followed by 8 frame/global attention pairs with width 384 and 6 attention heads.
The learning rate was 0.0006, with 10 warmup epochs and cosine decay to 5% of the initial rate. Weight decay was 0.01, and the camera and depth loss weights were 5 and 1, respectively.
Training Curves
Both training and validation losses decreased over time. The depth loss dropped quickly at the start, while the camera loss improved more gradually. Training losses fluctuated but showed a downward trend. At epoch 1,000, the training loss was 1.1532 and validation loss was 0.8635. We used the checkpoint with the lowest validation loss for the final evaluation.
Validation and Test Metrics
We evaluated the best checkpoint on all 28 validation scenes and all 28 test scenes, using 8 views from the first 32 prepared frames, image size 224, and seed 825. AUC values below range from 0 to 1.
| Split | Depth AbsRel ↓ | Depth δ1 ↑ |
|---|---|---|
| Validation | 0.1442 | 0.8099 |
| Test | 0.1578 | 0.8012 |
| Split | AUC Type | @5° ↑ | @15° ↑ | @30° ↑ |
|---|---|---|---|---|
| Validation | Rotation | 0.2207 | 0.5261 | 0.6994 |
| Validation | Translation | 0.0276 | 0.0925 | 0.2026 |
| Validation | Joint Pose | 0.0084 | 0.0676 | 0.1747 |
| Test | Rotation | 0.2265 | 0.5666 | 0.7443 |
| Test | Translation | 0.0099 | 0.0816 | 0.1946 |
| Test | Joint Pose | 0.0028 | 0.0676 | 0.1794 |
Qualitative Geometry Results
The following comparisons show ground-truth and predicted geometry for four different validation scenes. Each pair includes cameras, a point cloud, and the depth map for view 0. The viewer angles vary between screenshots, so we compare the scene structure and depth boundaries rather than exact screen positions.
Scene 1: 09c1414f1b
The room layout is recognizable, but the table and chair boundaries are less distinct in the predicted depth map.
Scene 2: 21d970d8de
The main floor and wall surfaces are recovered, while the table and chair boundary are smoother and not very clear.
Scene 3: 280b83fcf3
The hallway structure is preserved, but fine details near the shoes and doorway are smoothed out.
Scene 4: 2b1dc6d6a5
The office chair and surrounding surfaces remain recognizable, with noise and differences in the reconstructed surfaces.
Discussion
Test depth accuracy was slightly worse than validation: AbsRel increased from 0.1442 to 0.1578, and δ1 decreased from 0.8099 to 0.8012. This suggests some loss of depth accuracy on new scenes. Rotation AUC was higher on test, while translation AUC remained low on both splits. Estimating relative translation direction appears to be a larger challenge than estimating rotation. The low joint pose AUC indicates that accurate depth alone does not guarantee accurate multi-view camera geometry.
On these held-out scenes, the model recovers the main room and hallway structures, but predicted depth maps lose some small objects and sharp boundaries. The reconstructed surfaces also show noise and differences in shape. Both depth and camera errors can affect how well points from different views align.