Assignment 4: Multi-View Transformers

Tianle Lyu · Andrew ID: tianlel
16-825 Learning for 3D Vision

1. Multi-View Data Preprocessing (20 points)

1.1. Resizing Camera Intrinsics

The implementation is complete. Please refer to the submitted code for details.

1.2. Depth Unprojection

The implementation is complete. Please refer to the submitted code for details.
I unproject depth maps into 3D points using camera intrinsics and poses, visualizing the points and cameras in the original world coordinate system in meters.

1.3. Camera and Depth Canonicalization

I transform all camera poses into the first camera’s coordinate frame using \(T_i' = T_0^{-1}T_i\). I compute the scale as the distance from the reference origin to the valid-point centroid, then divide camera translations, depths, and 3D points by this scale without subtracting the centroid. This makes the first camera pose identity and the valid-point centroid distance equal to 1.

1.4. Camera Rays

I compute each pixel’s camera-space ray direction as \(K^{-1}[u,v,1]^T\), rotate it into the canonical world frame using \(R\), and normalize it to unit length. Each ray starts at the camera center \(o=t\); translation affects the origin, not the direction.

1.5. Plücker Coordinates

I compute the Plücker moment as \(m=o\times d\) and represent each ray using six channels \([d,m]\). The unit direction \(d\) describes the ray’s orientation, while the moment \(m\) encodes the supporting line’s position relative to the coordinate origin and remains unchanged when \(o\) moves along that line.

1.6. Understanding the Provided Dataloader

I understand how the provided dataloader works.

1.7. Final Check

I checked four training samples and confirmed the expected tensor shapes, point and ray reprojection errors below 0.01 pixels, and a normalized valid-point centroid distance of 1.0 for every sample.

2. Novel-View Synthesis with Mini-LVSM (40 points)

2.0. Implementation

The implementation is complete. Please refer to the submitted code for details.

2.1. Overfitting (10 points)

I trained Mini-LVSM on one fixed batch for 2,000 steps and achieved a final overfit PSNR of 30.95 dB. From left to right, the columns show source view 1, source view 2, the target ground truth, and the predicted target view.

2.2. Training and Evaluation (30 points)

Training curves

Ablation study

Source views Target-ray conditioning PSNR ↑ SSIM ↑
1 correct 15.22 0.5579
1 shuffled 11.80 0.4865
1 none 12.48 0.5275
2 correct 16.16 0.5599
2 shuffled 11.46 0.4705
2 none 12.45 0.5245
4 correct 16.92 0.5751
4 shuffled 11.56 0.4714
4 none 12.47 0.5339

In each visualization, the first S columns show the source views, followed by the target ground truth and the prediction. Each row corresponds to one target view.

Training visualization

Validation visualization at epoch 95

Evaluation visualization: Effect of the Number of Source Views

1 Source View (Correct Rays)
2 Source Views (Correct Rays)
4 Source Views (Correct Rays)

Evaluation visualization: Effect of Target-Ray Conditioning (2 Source Views)

Correct Target Rays
Shuffled Target Rays
Zero Target Rays (None)

Brief Summary

I trained Mini-LVSM for 100 epochs using 224 × 224 images, 8 views per sample, 2 source views, and a batch size of 4. I used AdamW with a learning rate of 2 × 10⁻⁴, weight decay of 0.05, and 200 warmup steps. The model used an 8 × 8 patch size, a hidden dimension of 384, and 8 transformer blocks. I enabled bfloat16 mixed precision and selected the checkpoint with the highest validation PSNR.

I evaluated the model with 1, 2, and 4 source views. With correct target rays, PSNR increased from 15.22 to 16.16 and 16.92 dB. This suggests that additional source views help, although the evaluated target sets differ across these settings.

I also compared correct, shuffled, and zero target rays. With two source views, shuffling the rays reduced PSNR to 11.46 dB, while removing them reduced it to 12.45 dB. These results indicate that the model uses target-ray information. However, the predictions remain noticeably blurry: correct rays produce coarse scene structure, shuffled rays produce mismatched layouts, and zero rays produce nearly uniform colors.

2.3. Bonus: Analysis and Exploration (10 points)

Experiment Question:

Does training with four source views improve novel-view synthesis compared with training with two source views?

Experiment Setup:

I investigated whether training Mini-LVSM with four source views improves novel-view synthesis compared with training with two source views. I trained both models from scratch for 100 epochs using the same remaining configuration described in Section 2.2. With eight total views per sample, the baseline used two source views and six targets, while the new model used four source views and four targets. I selected each model’s best checkpoint by validation PSNR and evaluated both with 1, 2, and 4 source views using correct target rays.

Metrics Comparison

Evaluation source views Train S=2 PSNR ↑ Train S=4 PSNR ↑ Train S=2 SSIM ↑ Train S=4 SSIM ↑
1 15.22 15.29 0.5579 0.5567
2 16.16 16.40 0.5599 0.5600
4 16.92 17.42 0.5751 0.5782

Visual Comparison: Training with 2 vs. 4 Source Views

All results use correct target rays. Within each image, the columns show the source views, target ground truth, and prediction.

Train: 2 Source Views · Evaluate: 1 Source View
Train: 4 Source Views · Evaluate: 1 Source View
Train: 2 Source Views
Evaluate: 2 Source Views
Train: 4 Source Views
Evaluate: 2 Source Views
Train: 2 Source Views · Evaluate: 4 Source Views
Train: 4 Source Views · Evaluate: 4 Source Views

Results

I found that training with four source views increased validation PSNR at all three evaluation source counts. The largest gain occurred at four evaluation source views, where PSNR increased from 16.92 to 17.42 dB and SSIM increased from 0.5751 to 0.5782. A possible explanation is that additional source views provide more scene information, and this evaluation setting matches the new model’s training source count. However, SSIM slightly decreased when evaluating with only one source view.

The predictions remained heavily blurred, with broad color and brightness patterns but few clear object boundaries or textures. Therefore, the numerical improvement did not resolve the main visual limitation.

This comparison also has limitations. Increasing the training source count from two to four reduced the number of supervised targets from six to four, and equal epoch counts do not imply equal computation. I used one run per setting, so small differences may reflect training variability. I compared the models at each fixed evaluation source count because the evaluated target sets differ across source counts.

3. Multi-View Geometry with Mini-VGGT (40 points)

3.0. Implementation

The implementation is complete. Please refer to the submitted code for details.

3.1. Overfitting (10 points)

Loss curve image

Ground Truth Geometry

Predicted Geometry

I overfit Mini-VGGT on one fixed training clip. The total, camera, and depth losses decreased during training. The predicted geometry recovered the main scene structure and camera arrangement, although some local depth errors and scattered points remained.

3.2. Training and Evaluation (30 points)

Experiment Setup:

I trained Mini-VGGT for 1,000 epochs using 224 × 224 images, eight views per sample, and a sampling range of 32. I used a configured batch size of 16, a learning rate of 6 × 10⁻⁴, weight decay of 0.01, and 10 warmup epochs. The model used a frozen DINOv2 ViT-B/14 encoder and eight frame/global attention pairs with a hidden dimension of 384 and six attention heads. I weighted the camera and depth losses by 5.0 and 1.0, respectively, and selected the checkpoint with the lowest validation loss.

Loss curve image

Validation/test Metrics

Metric Validation Test
Depth AbsRel ↓ 0.1427 0.1602
Depth δ1 ↑ 0.8367 0.7923
Rotation AUC @ 5° ↑ 0.1992 0.1796
Rotation AUC @ 15° ↑ 0.5297 0.4985
Rotation AUC @ 30° ↑ 0.7090 0.6861
Translation AUC @ 5° ↑ 0.0135 0.0110
Translation AUC @ 15° ↑ 0.0764 0.0531
Translation AUC @ 30° ↑ 0.1781 0.1536
Joint Pose AUC @ 5° ↑ 0.0069 0.0015
Joint Pose AUC @ 15° ↑ 0.0564 0.0320
Joint Pose AUC @ 30° ↑ 0.1543 0.1269

Comparison of GT and predicted geometry

Left: Ground-truth geometry. Right: Predicted geometry. Each row shows a different validation scene.

Brief Summary

I trained Mini-VGGT on the full training split with eight views per sample and a sampling range of 32. I selected the best checkpoint by validation loss and evaluated both complete validation and test splits using eight views from the first 32 prepared frames, an image size of 224 × 224, and seed 825.

I obtained a depth AbsRel of 0.1427 and δ1 of 0.8367 on validation, compared with 0.1602 and 0.7923 on test. Rotation AUC at 30° reached 0.7090 and 0.6861, respectively. Translation and joint pose AUC were substantially lower, indicating that relative translation direction remained a major limitation.

In the four validation scenes, I observed that the predictions captured the broad scene structure but produced smoother depth boundaries, distorted surfaces, and scattered or misaligned points. These errors are consistent with imperfect depth and camera predictions. Although training and validation losses decreased overall, the remaining geometric errors show that generalization to unseen scenes is still limited.

3.3. Bonus: Analysis and Exploration (10 points)

Experiment Question:

Does increasing the number of input views at evaluation improve the depth and camera estimation accuracy of Mini-VGGT trained with eight views?

Experiment Setup:

I investigated how the number of input views at evaluation affects the depth and camera estimation accuracy of Mini-VGGT trained with eight views. I used the same checkpoint selected by the lowest validation loss for all experiments, without retraining. I evaluated 2, 4, and 8 input views on all 28 validation scenes, using the first 32 prepared frames, a resolution of 224 × 224, and seed 825. I compared depth AbsRel, depth δ1, and rotation, translation, and joint pose AUC, alongside geometry visualizations. Changing the view count also changes the evaluated frames and camera pairs, so this comparison does not isolate the effect of additional context on a fixed set of predictions. Runtime and GPU memory were not measured.

Metrics Comparison

Evaluation views Depth AbsRel ↓ Depth δ1 ↑ Rotation
AUC@30° ↑
Translation
AUC@30° ↑
Joint Pose
AUC@30° ↑
2 0.1132 0.8748 0.7238 0.1774 0.1643
4 0.1346 0.8630 0.7204 0.1923 0.1615
8 0.1427 0.8367 0.7090 0.1781 0.1543

Each comparison shows one scene in a 2 × 2 layout: top left—ground-truth geometry from 8 views; top right—prediction using 2 views; bottom left—prediction using 4 views; bottom right—prediction using 8 views. I use the same trained checkpoint for all predictions and show the eight-view ground truth as a shared reference. Differences in point-cloud coverage may reflect the selected input frames as well as reconstruction errors.

Visualization Comparison 1:

Visualization Comparison 2:

Visualization Comparison 3:

Brief Discussion

I found that increasing the number of evaluation views did not consistently improve accuracy. Two views achieved the lowest depth AbsRel (0.1132), highest depth δ1 (0.8748), and highest rotation and joint pose AUC at 30°. Four views achieved the highest translation AUC at 30° (0.1923). Therefore, using eight views did not give the best results, even though the model was trained with eight views.

In the visualizations, additional views sometimes provided broader scene coverage, but distorted surfaces and misaligned points remained. Greater coverage does not necessarily indicate more accurate reconstruction. One possible explanation is that additional frames introduce challenging viewpoints and require the model to combine information across more views.

However, changing the view count also changes the evaluated frames, valid depth pixels, camera pairs, and scene normalization scale. These differences limit direct comparisons, so the results do not establish that fewer views are inherently better. A stronger follow-up would evaluate common frames and camera pairs with a shared normalization while varying only the additional input context. I used one checkpoint and one sampling seed, and did not measure runtime or GPU memory.