| Source views | Target-ray conditioning | PSNR ↑ | SSIM ↑ |
|---|---|---|---|
| 1 | correct | 15.22 | 0.5579 |
| 1 | shuffled | 11.80 | 0.4865 |
| 1 | none | 12.48 | 0.5275 |
| 2 | correct | 16.16 | 0.5599 |
| 2 | shuffled | 11.46 | 0.4705 |
| 2 | none | 12.45 | 0.5245 |
| 4 | correct | 16.92 | 0.5751 |
| 4 | shuffled | 11.56 | 0.4714 |
| 4 | none | 12.47 | 0.5339 |
In each visualization, the first S columns show the source views, followed by the target ground truth and the prediction. Each row corresponds to one target view.
I trained Mini-LVSM for 100 epochs using 224 × 224 images, 8 views per sample, 2 source views, and a batch size of 4. I used AdamW with a learning rate of 2 × 10⁻⁴, weight decay of 0.05, and 200 warmup steps. The model used an 8 × 8 patch size, a hidden dimension of 384, and 8 transformer blocks. I enabled bfloat16 mixed precision and selected the checkpoint with the highest validation PSNR.
I evaluated the model with 1, 2, and 4 source views. With correct target rays, PSNR increased from 15.22 to 16.16 and 16.92 dB. This suggests that additional source views help, although the evaluated target sets differ across these settings.
I also compared correct, shuffled, and zero target rays. With two source views, shuffling the rays reduced PSNR to 11.46 dB, while removing them reduced it to 12.45 dB. These results indicate that the model uses target-ray information. However, the predictions remain noticeably blurry: correct rays produce coarse scene structure, shuffled rays produce mismatched layouts, and zero rays produce nearly uniform colors.
Does training with four source views improve novel-view synthesis compared with training with two source views?
I investigated whether training Mini-LVSM with four source views improves novel-view synthesis compared with training with two source views. I trained both models from scratch for 100 epochs using the same remaining configuration described in Section 2.2. With eight total views per sample, the baseline used two source views and six targets, while the new model used four source views and four targets. I selected each model’s best checkpoint by validation PSNR and evaluated both with 1, 2, and 4 source views using correct target rays.
| Evaluation source views | Train S=2 PSNR ↑ | Train S=4 PSNR ↑ | Train S=2 SSIM ↑ | Train S=4 SSIM ↑ |
|---|---|---|---|---|
| 1 | 15.22 | 15.29 | 0.5579 | 0.5567 |
| 2 | 16.16 | 16.40 | 0.5599 | 0.5600 |
| 4 | 16.92 | 17.42 | 0.5751 | 0.5782 |
All results use correct target rays. Within each image, the columns show the source views, target ground truth, and prediction.
I found that training with four source views increased validation PSNR at all three evaluation source counts. The largest gain occurred at four evaluation source views, where PSNR increased from 16.92 to 17.42 dB and SSIM increased from 0.5751 to 0.5782. A possible explanation is that additional source views provide more scene information, and this evaluation setting matches the new model’s training source count. However, SSIM slightly decreased when evaluating with only one source view.
The predictions remained heavily blurred, with broad color and brightness patterns but few clear object boundaries or textures. Therefore, the numerical improvement did not resolve the main visual limitation.
This comparison also has limitations. Increasing the training source count from two to four reduced the number of supervised targets from six to four, and equal epoch counts do not imply equal computation. I used one run per setting, so small differences may reflect training variability. I compared the models at each fixed evaluation source count because the evaluated target sets differ across source counts.
I trained Mini-VGGT for 1,000 epochs using 224 × 224 images, eight views per sample, and a sampling range of 32. I used a configured batch size of 16, a learning rate of 6 × 10⁻⁴, weight decay of 0.01, and 10 warmup epochs. The model used a frozen DINOv2 ViT-B/14 encoder and eight frame/global attention pairs with a hidden dimension of 384 and six attention heads. I weighted the camera and depth losses by 5.0 and 1.0, respectively, and selected the checkpoint with the lowest validation loss.
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1427 | 0.1602 |
| Depth δ1 ↑ | 0.8367 | 0.7923 |
| Rotation AUC @ 5° ↑ | 0.1992 | 0.1796 |
| Rotation AUC @ 15° ↑ | 0.5297 | 0.4985 |
| Rotation AUC @ 30° ↑ | 0.7090 | 0.6861 |
| Translation AUC @ 5° ↑ | 0.0135 | 0.0110 |
| Translation AUC @ 15° ↑ | 0.0764 | 0.0531 |
| Translation AUC @ 30° ↑ | 0.1781 | 0.1536 |
| Joint Pose AUC @ 5° ↑ | 0.0069 | 0.0015 |
| Joint Pose AUC @ 15° ↑ | 0.0564 | 0.0320 |
| Joint Pose AUC @ 30° ↑ | 0.1543 | 0.1269 |
Left: Ground-truth geometry. Right: Predicted geometry. Each row shows a different validation scene.
I trained Mini-VGGT on the full training split with eight views per sample and a sampling range of 32. I selected the best checkpoint by validation loss and evaluated both complete validation and test splits using eight views from the first 32 prepared frames, an image size of 224 × 224, and seed 825.
I obtained a depth AbsRel of 0.1427 and δ1 of 0.8367 on validation, compared with 0.1602 and 0.7923 on test. Rotation AUC at 30° reached 0.7090 and 0.6861, respectively. Translation and joint pose AUC were substantially lower, indicating that relative translation direction remained a major limitation.
In the four validation scenes, I observed that the predictions captured the broad scene structure but produced smoother depth boundaries, distorted surfaces, and scattered or misaligned points. These errors are consistent with imperfect depth and camera predictions. Although training and validation losses decreased overall, the remaining geometric errors show that generalization to unseen scenes is still limited.
Does increasing the number of input views at evaluation improve the depth and camera estimation accuracy of Mini-VGGT trained with eight views?
I investigated how the number of input views at evaluation affects the depth and camera estimation accuracy of Mini-VGGT trained with eight views. I used the same checkpoint selected by the lowest validation loss for all experiments, without retraining. I evaluated 2, 4, and 8 input views on all 28 validation scenes, using the first 32 prepared frames, a resolution of 224 × 224, and seed 825. I compared depth AbsRel, depth δ1, and rotation, translation, and joint pose AUC, alongside geometry visualizations. Changing the view count also changes the evaluated frames and camera pairs, so this comparison does not isolate the effect of additional context on a fixed set of predictions. Runtime and GPU memory were not measured.
| Evaluation views | Depth AbsRel ↓ | Depth δ1 ↑ | Rotation AUC@30° ↑ |
Translation AUC@30° ↑ |
Joint Pose AUC@30° ↑ |
|---|---|---|---|---|---|
| 2 | 0.1132 | 0.8748 | 0.7238 | 0.1774 | 0.1643 |
| 4 | 0.1346 | 0.8630 | 0.7204 | 0.1923 | 0.1615 |
| 8 | 0.1427 | 0.8367 | 0.7090 | 0.1781 | 0.1543 |
Each comparison shows one scene in a 2 × 2 layout: top left—ground-truth geometry from 8 views; top right—prediction using 2 views; bottom left—prediction using 4 views; bottom right—prediction using 8 views. I use the same trained checkpoint for all predictions and show the eight-view ground truth as a shared reference. Differences in point-cloud coverage may reflect the selected input frames as well as reconstruction errors.
I found that increasing the number of evaluation views did not consistently improve accuracy. Two views achieved the lowest depth AbsRel (0.1132), highest depth δ1 (0.8748), and highest rotation and joint pose AUC at 30°. Four views achieved the highest translation AUC at 30° (0.1923). Therefore, using eight views did not give the best results, even though the model was trained with eight views.
In the visualizations, additional views sometimes provided broader scene coverage, but distorted surfaces and misaligned points remained. Greater coverage does not necessarily indicate more accurate reconstruction. One possible explanation is that additional frames introduce challenging viewpoints and require the model to combine information across more views.
However, changing the view count also changes the evaluated frames, valid depth pixels, camera pairs, and scene normalization scale. These differences limit direct comparisons, so the results do not establish that fewer views are inherently better. A stronger follow-up would evaluate common frames and camera pairs with a shared normalization while varying only the additional input context. I used one checkpoint and one sampling seed, and did not measure runtime or GPU memory.