Assignment 4: Multi-View Transformers

16-825: Learning for 3D Vision · Shohei Nagai (Andrew ID: shoheina)

1. Multi-View Data Preprocessing

Depth Unprojection

Depth from all eight sampled views is unprojected with the resized intrinsics and the original camera-to-world poses, so the points are in the dataset's metric world frame.

Unprojection, scene 036bce3393

*Figure 1: Unprojection for scene 036bce3393. (Red frustum: camera0, blue frustums: the other seven sampled views.)

Unprojection, scene 2Unprojection, scene 3Unprojection, scene 4Unprojection, scene 5
Figure 2: Unprojection for the remaining four scenes.

Camera and Depth Canonicalization

All cameras and points are expressed in camera 0's frame, then divided by the distance of the valid-point centroid, so every sample shares the same frame and scale. The first camera center and valid-point centroid distance are described in the right pane of each screenshot.

First camera center: (0.0000, 0.0000, 0.0000) · Valid-point centroid distance: 1.000 · Scene scale (scene 036bce3393): 1.867 m
Canonicalization, scene 036bce3393

*Figure 3: Canonicalization for scene 036bce3393. All cameras and points are expressed in camera 0's frame. (Red frustum: camera0, blue frustums: the other seven sampled views.)

Canonicalization, scene 2Canonicalization, scene 3Canonicalization, scene 4Canonicalization, scene 5
Figure 4: Canonicalization for the remaining four scenes. All cameras and points are expressed in camera 0's frame.

Canonicalization: Every pose is multiplied by the inverse of camera 0's pose, so camera 0 becomes the identity and the other cameras are expressed relative to it. Depth is unprojected in this frame, and translations, depth and points are all divided by one scene scale, which is the distance from camera 0 to the centroid of the valid points, which ends up at distance 1.

World-space ray directions: Each image pixel corresponds to a ray starting at the camera center and passing through that pixel. The world-space ray direction is this direction after transforming it from the camera's local coordinate system into the shared 3D world coordinate. Using world space lets the model compare rays from different camera views consistently.

Plücker moment: Plücker moment is a 6D encoding (d, m) which uniquely describes a 3D line relative to the world origin. d is the unit direction and m is the moment m = o x d where o is the origin. The moment is invariant to sliding the origin along the line.

Final Check

All four training samples pass. Reprojection errors are much below the 0.01 px threshold and the valid-point centroid stays at distance 1 from camera 0.

Max reprojection error: 1.2e-4 px (points), 1.0e-4 px (rays) · Centroid distance: 1.000 · Valid depth: 75%-99%
Full output of python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010810750973178074,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.00010235906665911898,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 8.28983829705976e-05,
      "max_ray_reprojection_error_px": 7.938443741295487e-05,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 9.5367431640625e-05,
      "max_ray_reprojection_error_px": 8.996945689432323e-05,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

I implemented the relevant code and confirmed the implemented model can memorize one fixed batch. The final overfit PSNR and the val_epoch000.png visualization are as follows:

Final overfit PSNR: 30.84 dB
LVSM overfit visualization
Figure 5: Overfitting one batch. (Columns: source views, ground truth, prediction.)

2.2 Training and Evaluation

I implemented the relevant code and trained on the full training split with the training settings below. I show the training curves in Fig. 6, validation PSNR/SSIM for each ablation setting in Fig. 7, and the training and evaluation visualization

Training settings. I used the settings from configs/lvsm.json (shown below) without changes.

SettingValue
Epochs100
Source / target views1,2,4 / 6
Batch size4
OptimizerAdamW
Learning rate0.0002
Weight decay0.05
Betas(0.9 0.95)
LossMSE on target pixels
Warmup steps200
Gradient clipping1.0
Table 1: Training settings for Mini-LVSM
LVSM training curves
Figure 6: Training and validation MSE (left) and PSNR (right).
Source viewsConditioningPSNR ↑SSIM ↑
1correct15.180.558
1shuffled11.810.488
1none12.470.530
2correct16.130.561
2shuffled11.500.473
2none12.450.527
4correct16.860.575
4shuffled11.600.474
4none12.460.536
Table 2: Validation PSNR/SSIM for every combination of source-view count and ray conditioning.
Training-time visualization, epoch 95
Figure 7: Training visualizations at epoch 95 (Columns: 2 source views, ground truth, prediction)
S=1, correctS=2, correct
Figure 8: Evaluation visualization with 1 (left) and 2 (right) source views and correct target rays. (Columns: source views, ground truth, prediction)
S=4, correctS=4, shuffledS=4, none
Figure 9: Evaluation with 4 source views and correct (left), shuffled (middle) and zero (right) target rays. (Columns: source views, ground truth, prediction)

Effect of the number of source views. The validation PSNR/SSIM in the Table 2 shows more source views improve both PSNR and SSIM. PSNR rises from 15.18 dB with one view to 16.86 dB with four, and SSIM increases from 0.558 to 0.575. Extra views cover more of the target view and give the model parallax to triangulate from, where as a single source forces it to guess the geometry and give a blurrier result. The gain carries over to four sources even though training used two because all sources share the same embedding and projection.

Effect of ray conditioning. The target rays are the only information about the target viewpoint, and the model depends on them completely. With shuffled rays, PSNR drops by about 5dB at every numbers of source views and render an image of the wrong view (Fig. 9, middle). With zero rays, all target tokens are identical so the output collapses to the scene's mean color (Fig. 9, right). The scores slightly higher in PSNR than shuffled because a blurry mean outperforms a sharp but wrong view under MSE. With no ray, more sources no longer help in either PSNR or SSIM.

2.3 Bonus: Analysis and Exploration — Training View Count

Question

The baseline is trained with a fixed S = 2 source views but evaluated with S = 1, 2 and 4. How does training with a different number of source views, or a random number per step, change the evaluation with 1 to 6 source views?

Hypothesis is that a fixed-S model degrades away from its training S because of the sequence-length mismatch, and that training with a random S fixes this.

Experimental settings

I trained five models with the Q2.2 settings (100 epochs, batch size 4, same optimizer, schedule, seed, model and data) and changed only the training source view count: fixed S = 1, 2 (Q2.2 baseline), 4 and 6, and random S ~ U{1, 2, 4} drawn per step. Since each clip has 8 views, a larger S also means fewer targets (T = 8 − S). Each best-validation-PSNR checkpoint is evaluated on the validation split with S = 1 to 6 source views and correct target rays.

Results

The training view count matters by up to about 3.0 dB in PSNR and 0.02 in SSIM at the same evaluation S (Figure 11). Every multi-source model improves from evaluation S = 1 to S = 5, and the model trained with S = 4 is best or tied-best at every evaluation S. The S = 1 model gets worse with every added source, and the random-S model performs the same as the fixed S = 2 baseline. All models, including the one trained with S = 6, drop at evaluation S = 6. Figure 12 shows the gap visually: the S = 4 model draws more detailed and accurate predictions than the S = 1 model.

Training curves for different training S
Figure 10: Train (left) and validation (right) PSNR during training. Each run is validated at its own training S (the random-S run at S = 2).
PSNR and SSIM against the number of evaluation source views
Figure 11: Validation PSNR (left) and SSIM (right) with correct target rays against the number of source views at evaluation.
Model trained with S=1, evaluated with S=4Model trained with S=4, evaluated with S=4
Figure 12: Models trained with S = 1 (left) and S = 4 (right), both evaluated with 4 source views on the same scenes. (Columns: 4 sources, ground truth, prediction.)

Conclusions

Training view count matters, but not because of a sequence-length mismatch, since the S = 4 model wins even at the lengths it never saw and random S gives nothing over the baseline. Performance improves as the training S increases up to S = 4 and then drops at S = 6. For this 8-view setup the best setting is to train with S = 4 and evaluate with as many sources as are available.

Possible explanations and limitations

More sources help because they cover more of the target and give the model parallax, but with 8 views per clip, S = 6 leaves only 2 targets per sample, so supervision drops to a third of S = 2. Random S adds nothing because sources are an unordered set in this architecture (shared positional embedding, no view-index embedding), so what is learned at one S transfers to any other. The S = 1 model never learned to choose between sources, so extra views only confuse it. The main limitations are a single seed per setting, short 100-epoch runs, and an evaluation target set that changes with S, which makes the drop at S = 6 partly an evaluation set effect.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

I overfit one fixed training clip with the default Mini-VGGT and shows the loss curve (total, camera and depth loss) in Fig. 13 and the screenshots comparing GT and prediction in Fig. 14.

VGGT overfit loss curve
Figure 13: Overfit loss curves: total (left), camera (middle) and depth (right).
GT geometry, overfitPredicted geometry, overfit
Figure 14: Ground truth (left) and prediction (right) on the overfit clip.
Final overfit metrics: AbsRel 0.037 · δ1 0.956 · rotation error 0.16° · translation-direction error 0.49° · joint pose AUC@5/15/30 = 0.96 / 0.99 / 0.99

3.2 Training and Evaluation

Training settings. I used the settings from configs/vggt.json (shown below) without changes.

SettingValue
Epochs1000
Steps per epoch14
Batch size16
Loss5 · camera + 1 · depth
OptimizerAdamW
Learning rate0.0006
Weight decay0.01
Betas(0.9 0.95)
Warmup steps10
Gradient clipping1.0
VGGT training loss curve
Figure 15: Training (blue) and validation (orange) loss per epoch: total (left), camera (middle) and depth (right).
SplitAbsRel ↓δ1 ↑Rot AUC@5Rot AUC@15Rot AUC@30Trans AUC@5Trans AUC@15Trans AUC@30Joint AUC@5Joint AUC@15Joint AUC@30
Val0.1520.7970.2380.5740.7420.0170.0810.1860.0070.0660.168
Test0.1690.7630.2310.5690.7480.0080.0690.1920.0030.0510.174
Table 3: Validation and test metrics of best.pt.
Scene 1, GTScene 1, prediction
Figure 16: Validation scene 09c1414f1b, ground truth (left) and prediction (right).
Scene 2, GTScene 2, prediction
Figure 17: Validation scene 21d970d8de, ground truth (left) and prediction (right).
Scene 3, GTScene 3, prediction
Figure 18: Validation scene 280b83fcf3, ground truth (left) and prediction (right).
Scene 4, GTScene 4, prediction
Figure 19: Validation scene 2b1dc6d6a5, ground truth (left) and prediction (right).

Errors on held-out scenes. Depth transfers well to new rooms with AbsRel 0.15 for validation and 0.17 for test, and δ1 is around 0.8 for both. Its errors are the smooth inflated surfaces, discontinuities and missing thin structures seen in Fig. 16-19, which is expected from a 14px patch-level prediction. Rotation performs reasonably well (8.7° mean pairwise error), but translation direction is poor (40° mean pairwise error) and pulls the joint pose AUC@30 down to 0.17. The camera baselines in clips are short so a small absolute translation error becomes a large angular error. The training camera loss is still falling at epoch 1000 while validation has flattened, so more epochs or a larger translation weight would help improve the pose metrics.

3.3 Bonus: Analysis and Exploration — Point-Map Prediction

Question

I explore point-map prediction. The baseline gets points by unprojecting predicted depth with predicted cameras, so camera errors move whole depth maps. My question is: does the direct point-map prediction head give better point predictions than the camera + depth?

Experimental settings

I added a PointHead that copies the depth head with 3 output channels (dense linear prediction, RGB-guided refinement, patch-level confidence) and a point_loss that mirrors the depth loss with the per-pixel Euclidean distance to the GT points as the error. The total loss is 5 camera + 1 depth + 1 points. Everything else is identical to Q3.2. For every GT-valid pixel I compare two point reconstructions: predicted cameras + predicted depth (the baseline route) and the direct point head.

Results

The direct point head does not beat the camera + depth of the same model: 0.231 vs 0.225 on val and 0.218 vs 0.212 on test, about 2–3 % worse, with the same ranking in every view (Figure 20, right). Even in view 0, where there is no camera error to bypass, the head is slightly worse (0.125 vs 0.116). Adding the point loss does help the other tasks. The joint model improves depth AbsRel from 0.152 to 0.138 in validation and 0.169 to 0.151 in test, its own camera + depth points by 0.044 in validation and 0.024 in test, and joint pose AUC@30 from 0.17 to 0.18 in validation and 0.20 in test. Figures 21-22 show the two reconstructions are nearly identical in layout, with the point-head cloud slightly more scattered.

ModelSplitPoint error: cam + depthPoint headAbsRel ↓δ1 ↑Rot err ↓Rot AUC@30Trans AUC@30Joint AUC@30
camera + depth (Q3.2)val0.269 (0.54 m)—0.1520.79711.0°0.7420.1860.168
+ point headval0.225 (0.45 m)0.231 (0.47 m)0.1380.8429.4°0.7480.1960.181
camera + depth (Q3.2)test0.236 (0.47 m)—0.1690.7638.7°0.7480.1920.174
+ point headtest0.212 (0.41 m)0.218 (0.42 m)0.1510.8329.2°0.7420.2160.201
Table 4: Mean per-pixel point error of two reconstruction routes in normalized units and standard Q3.2 metrics.
Point error by reconstruction route and by view
Figure 20: Validation point error of two routes for both models (left) and per view index (right). View 0 is the reference camera, so its error is only depth error.
Scene 1, GTScene 1, cameras + depthScene 1, point head
Figure 21: Validation scene 09c1414f1b with the joint model: GT (left), points from predicted cameras + depth (middle) and the direct point head (right).
Scene 4, GTScene 4, cameras + depthScene 4, point head
Figure 22: Validation scene 2b1dc6d6a5: GT (left), points from predicted cameras + depth (middle) and the direct point head (right).

Conclusions

A direct point head is not a better way to get points in this setup. The best points come from the joint model's cameras + depth while the direct model is close second. The point loss is nevertheless worth adding as an auxiliary task, since it improves depth, points and pose at once for more time per epoch.

Possible explanations and limitations

Both heads read the same tokens, so the point head only knows the pose that is already in them and can't beat the dedicated camera head. Regressing absolute coordinates per patch also adds noise. The point loss helps the other tasks because it is the only term that couples pose and depth, pushing the shared features towards a pose and depth that agree. The limitations are one seed per model, the simple point head, and the fact that camera error dominates the point error for both methods, so a better camera head would matter more than any change to the point head.