16-825 Assignment 4: Multi-View Transformers

Joshua Tsai | Andrew ID: joshuat3

Q1. Multi-View Data Preprocessing (20 points)

1. Depth Unprojection

We unprojected the depth maps into 3D point clouds using the camera intrinsics and transformed the points into world coordinates using the camera poses. The visualization below shows the reconstructed scene from eight sampled views in the original world coordinate system.

Depth unprojection visualization
Figure 1. Depth unprojection in the original world coordinate system. The first camera center is at (3.3463, 4.6765, 1.4680) m, and the valid-point centroid distance is 6.668 m.

2. Camera Canonicalization

We use the first camera as the reference frame and scale the camera positions, depth, and 3D points so the valid-point centroid is 1 unit from the origin.

Camera canonicalization visualization
Figure 2. Canonicalized scene with the first camera at the origin (0, 0, 0). The scene scale is 1.867 m, and the valid-point centroid distance is 1.000 normalized units.

3. World-Space Ray Directions

We use the camera intrinsics to convert each pixel into a ray direction, rotate it into world coordinates, and normalize it to unit length.

4. Plücker Moment

The moment is computed as m = o × d and describes the line's position relative to the origin. We combine it with the direction to form six channels, [d, m].

5. Data Sanity Check

We checked four training samples using check_data.py. The tensor shapes matched the specification, and valid depth ratios ranged from 75.2% to 99.0%. The maximum point and ray reprojection errors were approximately 0.000123 and 0.000111 pixels, respectively, both below the required 0.01 pixels. The centroid distance was 1.0 for all samples, as expected.

Metric Result
Samples checked 4
Tensor shapes Passed
Valid depth ratio 75.2% - 99.0%
Max point reprojection error 0.000123 px
Max ray reprojection error 0.000111 px
Centroid distance 1.000

Q2. Novel-View Synthesis with Mini-LVSM (40 points)

1. Overfitting

We trained the Mini-LVSM model on a fixed batch for 2,000 steps to verify that the model could overfit a small set of training examples. The model contains 14.6 million parameters and uses two source views.

After 2,000 steps, the model achieved a validation PSNR of 30.92 dB and an SSIM of 0.8703, meeting the target PSNR of approximately 30 dB. The final training loss was 0.00684, and the validation loss was 0.00087. Training took approximately 2.11 hours on the GPU.

Metric Result
Training Steps 2,000
Source Views 2
Train PSNR 25.40 dB
Validation PSNR 30.92 dB
Validation SSIM 0.8703
Training Time 7,580 s (2.11 hours)
Mini-LVSM overfitting results
Figure 3. Mini-LVSM overfitting results after 2,000 training steps. The model reconstructs the main scene structures, including tables, chairs, and walls, although some predictions remain blurry and contain visible artifacts.

Overall, the overfitting experiment confirms that the Mini-LVSM implementation can learn the mapping from source views and target rays to the target RGB image.

2. Full Training and Evaluation

Training Results

We trained Mini-LVSM for 100 epochs with 2 source views, batch size 4, and learning rate 2e-4. Training took about 6.2 hours.

The training loss decreased from 0.06344 to 0.03394. The best validation PSNR was 16.13 dB at epoch 97, and the best SSIM was 0.5600 at epoch 81.

MetricResult
Epochs100
Batch Size4
Source Views2
Learning Rate2e-4
Best Validation PSNR16.13 dB
Best Validation SSIM0.5600
Training Time6.2 hours

Training Curves

Both training and validation losses decreased, while PSNR and SSIM improved over time. The curves became more stable toward the end, showing that the model was converging.

Training and validation loss.
Training and validation PSNR.
Validation SSIM.

Training Visualizations

The predictions improved as training progressed. The model learned the main scene structure, but some details remained blurry.

Epoch 0: Large missing or uniform regions.
Epoch 50: Main objects become more recognizable.
Epoch 95: Better reconstruction, but still blurry.

Evaluation and Ablation Study

We evaluated the model using 1, 2, and 4 source views with correct, shuffled, and no ray conditioning. The table shows PSNR (dB) / SSIM.

Source Views Correct Shuffled None
1 15.23 / 0.5568 11.77 / 0.4853 12.48 / 0.5267
2 16.13 / 0.5591 11.46 / 0.4718 12.46 / 0.5262
4 16.89 / 0.5748 11.57 / 0.4732 12.48 / 0.5362

The evaluation used 196, 168, and 112 images for 1, 2, and 4 source views, respectively.

Qualitative Evaluation

One Source View
Correct rays: 15.23 dB / 0.5568 SSIM.
Shuffled rays: 11.77 dB / 0.4853 SSIM.
No rays: 12.48 dB / 0.5267 SSIM.
Two Source Views
Correct rays: 16.13 dB / 0.5591 SSIM.
Shuffled rays: 11.46 dB / 0.4718 SSIM.
No rays: 12.46 dB / 0.5262 SSIM.
Four Source Views
Correct rays: 16.89 dB / 0.5748 SSIM.
Shuffled rays: 11.57 dB / 0.4732 SSIM.
No rays: 12.48 dB / 0.5362 SSIM.

Discussion

With correct ray conditioning, using more source views improved PSNR from 15.23 dB to 16.89 dB. The predictions also preserved more scene details.

Shuffled or missing rays significantly reduced reconstruction quality, even with more source views. This shows that correct camera ray information is important for novel-view synthesis.

Q3. Multi-View Geometry with Mini-VGGT (40 points)

1. Overfitting

We trained Mini-VGGT on one fixed training clip for 40 epochs, with 25 steps per epoch (1,000 steps in total).

Loss Curves

The total, camera, and depth losses all decreased substantially and became more stable toward the end of training.

Mini-VGGT overfit total, camera, and depth loss curves over 40 epochs
Mini-VGGT overfit loss curves.

Ground Truth and Predicted Geometry

The predicted camera arrangement and main room structure are close to the ground truth. Some noise and smoother depth boundaries remain in the prediction. The first predicted camera center is approximately (0.0000, 0.0001, -0.0001), close to the origin. The predicted valid-point centroid distance is 0.997, compared with 1.000 for the ground truth.

Ground-truth cameras, point cloud, and depth for the fixed training clip
Ground-truth geometry.
Predicted cameras, point cloud, and depth after single-clip overfitting
Predicted geometry after 40 epochs.

2. Full Training and Evaluation

Training Settings

We trained Mini-VGGT on all 224 training scenes for 1,000 epochs, using batch size 8 and 28 steps per epoch. Each sample contains 8 views sampled within a range of 32 frames at 224 × 224 resolution. We used a frozen DINOv2 ViT-B/14 encoder, followed by 8 frame/global attention pairs with width 384 and 6 attention heads.

The learning rate was 0.0006, with 10 warmup epochs and cosine decay to 5% of the initial rate. Weight decay was 0.01, and the camera and depth loss weights were 5 and 1, respectively.

Training Curves

Both training and validation losses decreased over time. The depth loss dropped quickly at the start, while the camera loss improved more gradually. Training losses fluctuated but showed a downward trend. At epoch 1,000, the training loss was 1.1532 and validation loss was 0.8635. We used the checkpoint with the lowest validation loss for the final evaluation.

Full-training and validation total, camera, and depth loss over 1000 epochs
Full-training loss curves. Camera and depth losses are unweighted.

Validation and Test Metrics

We evaluated the best checkpoint on all 28 validation scenes and all 28 test scenes, using 8 views from the first 32 prepared frames, image size 224, and seed 825. AUC values below range from 0 to 1.

SplitDepth AbsRel ↓Depth δ1 ↑
Validation0.14420.8099
Test0.15780.8012
SplitAUC Type@5° ↑@15° ↑@30° ↑
ValidationRotation0.22070.52610.6994
ValidationTranslation0.02760.09250.2026
ValidationJoint Pose0.00840.06760.1747
TestRotation0.22650.56660.7443
TestTranslation0.00990.08160.1946
TestJoint Pose0.00280.06760.1794

Qualitative Geometry Results

The following comparisons show ground-truth and predicted geometry for four different validation scenes. Each pair includes cameras, a point cloud, and the depth map for view 0. The viewer angles vary between screenshots, so we compare the scene structure and depth boundaries rather than exact screen positions.

Scene 1: 09c1414f1b

The room layout is recognizable, but the table and chair boundaries are less distinct in the predicted depth map.

Ground truth cameras, point cloud, and depth for scene 09c1414f1b
Scene 1: Ground truth.
Prediction cameras, point cloud, and depth for scene 09c1414f1b
Scene 1: Prediction.
Scene 2: 21d970d8de

The main floor and wall surfaces are recovered, while the table and chair boundary are smoother and not very clear.

Ground truth cameras, point cloud, and depth for scene 21d970d8de
Scene 2: Ground truth.
Prediction cameras, point cloud, and depth for scene 21d970d8de
Scene 2: Prediction.
Scene 3: 280b83fcf3

The hallway structure is preserved, but fine details near the shoes and doorway are smoothed out.

Ground truth cameras, point cloud, and depth for scene 280b83fcf3
Scene 3: Ground truth.
Prediction cameras, point cloud, and depth for scene 280b83fcf3
Scene 3: Prediction.
Scene 4: 2b1dc6d6a5

The office chair and surrounding surfaces remain recognizable, with noise and differences in the reconstructed surfaces.

Ground truth cameras, point cloud, and depth for scene 2b1dc6d6a5
Scene 4: Ground truth.
Prediction cameras, point cloud, and depth for scene 2b1dc6d6a5
Scene 4: Prediction.

Discussion

Test depth accuracy was slightly worse than validation: AbsRel increased from 0.1442 to 0.1578, and δ1 decreased from 0.8099 to 0.8012. This suggests some loss of depth accuracy on new scenes. Rotation AUC was higher on test, while translation AUC remained low on both splits. Estimating relative translation direction appears to be a larger challenge than estimating rotation. The low joint pose AUC indicates that accurate depth alone does not guarantee accurate multi-view camera geometry.

On these held-out scenes, the model recovers the main room and hallway structures, but predicted depth maps lose some small objects and sharp boundaries. The reconstructed surfaces also show noise and differences in shape. Both depth and camera errors can affect how well points from different views align.