16-825 Learning for 3D Vision

Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

1.2Depth Unprojection

Data visualization after unprojection:

Multi-view data visualization after depth unprojection

1.3Camera and Depth Canonicalization

Data visualization after canonicalization:

Multi-view data visualization after canonicalization

Canonicalization: A scene's coordinates depend on an arbitrary origin and scale, so raw values vary widely between scenes. Canonicalization describes every scene from the same starting point at the same size. It take camera 0 as the reference and compute every pose relative to it:

Ti′=T0−1*TiT_i' = T_0^{-1} * T_i

The equation above make T0′T_0' an identity matrix. After that, the average position for all valid 3D points, which is the scene's centroid, is obtained. The scene scale ss is the distance from camera 0 to the centroid of the valid points, computed in this new frame. Camera translations, depth and points are all divided by ss, so the centroid ends up at distance 1 from the origin. Rotations do not change.

1.4Camera Rays

World-space ray directions: Each pixel (u,v)(u,v) in the images correspond to a ray leaving the camera center, multiplying its homogeneous coordinates by the inverse intrinsics of the camera gives the direction in camera space:

dcam=K−1*[u,v,1]⊤d_{cam} = K^{-1} * [u,v,1]^\top

Rotating dcamd_{cam} by the camera-to-world rotation matrix and normalizing it yield the unit world-space direction:

d=R*dcam/||R*dcam||d = R * d_{cam} / ||R * d_{cam}||

Only the rotation RR is applied because a direction has no position. The translation gives the ray origin instead.

1.5Plücker Coordinates

Plücker Moment: If explaining the ray solely by its direction dd, it only gives the orientation of the ray so parallel rays will be identical. To solve this, a Plücker Moment mm is added to describe the difference:

[d,m]m=o×d[d,m] \qquad m = o \times d

Here, oo is the ray origin, mm is perpendicular to the plane containing the ray and the origin, and since dd is a unit vector, ||m||||m|| is the distance from the origin to the ray's line.

1.7Final Check

check_data.py output:

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.1Overfitting

Train PSNR 25.37dB
Validation PSNR 30.89dB
LVSM single-scene overfitting validation output
val_epoch000.png

2.2Training and Evaluation

Training Settings

SettingValue
Batch size4
Eval frames32
Patch size8
d_model384
Layers8
Head dim64
MLP ratio4.0
QK normtrue
Epochs100
Learning rate2e-4
Weight decay0.05
Betas(0.9, 0.95)
Warmup steps200
Grad clip1.0
AMPtrue
Source views2

Training curves

LVSM training curves

Validation metrics

The table below used PSNR(dB)/SSIM format

Source viewsCorrectShuffledNone
115.20 / 0.55811.80 / 0.48712.48 / 0.528
216.13 / 0.56011.49 / 0.47212.46 / 0.527
416.87 / 0.57511.59 / 0.47312.48 / 0.536

Target images evaluated: 196 for S=1, 168 for S=2, 112 for S=4.

Training visualizations

LVSM validation output at epoch 0
Epoch 0
LVSM validation output at epoch 50
Epoch 50
LVSM validation output at epoch 95
Epoch 95

Evaluation visualizations

S=1
1 source view, correct rays
Correct rays
1 source view, shuffled rays
Shuffled rays
1 source view, no rays
No rays
S=2
2 source views, correct rays
Correct rays
2 source views, shuffled rays
Shuffled rays
2 source views, no rays
No rays
S=4
4 source views, correct rays
Correct rays
4 source views, shuffled rays
Shuffled rays
4 source views, no rays
No rays

Effect of the number of source views: When the model gets the correct rays, adding source views helps steadily. PSNR climbs from 15.20 dB with 1 view to 16.87 dB with 4, and SSIM goes from 0.558 to 0.575. This is because each extra view shows more of the scene, so the model has less to predict on its own.

Effect of ray conditioning: The effect of rays is huge. If the ray is taken away, PSNR falls to roughly 11.5–12.5 dB no matter how many source views we provide, so the benefit of extra views vanishes too. Without rays, the model has no way of knowing which viewpoint we're asking for, so it hedges and produces nearly the same blurry, averaged image every time. Shuffled rays are even worse than no rays. The model confidently renders a view for the wrong pose, which under MSE a this costs more than a vague average.

3. Multi-View Geometry with Mini-VGGT

3.1Overfitting

Loss Curve

VGGT single-scene overfitting loss curve
Ground-truth scene for the overfitting run
GT Scene
Predicted scene for the overfitting run
Predicted Scene

3.2Training and Evaluation

Training Settings

SettingValue
Views / sampling range8 / 32
Image size224
Batch size16 (14 steps per epoch)
EncoderDINOv2 ViT-B/14 (frozen)
Embedding dim / heads / layers384 / 6 / 8
Point headOff
Epochs1000
OptimizerAdamW
Learning rate6e-4, decaying to 3e-5
Warmup epochs10
Weight decay0.01
Grad clip1.0
Loss weightsCamera 5.0, depth 1.0
Precisionbf16
GPUs1

Loss Curve

VGGT training loss curve

Metrics

MetricValidationTest
Depth AbsRel0.1380.157
Depth δ10.8500.809
Rotation AUC @ 5°0.2080.178
Rotation AUC @ 15°0.5220.505
Rotation AUC @ 30°0.7020.712
Translation AUC @ 5°0.0180.013
Translation AUC @ 15°0.0960.058
Translation AUC @ 30°0.2050.159
Joint Pose AUC @ 5°0.0070.004
Joint Pose AUC @ 15°0.0700.036
Joint Pose AUC @ 30°0.1780.141

GT vs. Predicted Geometry

Frame 1
Ground-truth geometry, frame 1
GT
Predicted geometry, frame 1
Predicted
Frame 2
Ground-truth geometry, frame 2
GT
Predicted geometry, frame 2
Predicted
Frame 3
Ground-truth geometry, frame 3
GT
Predicted geometry, frame 3
Predicted
Frame 4
Ground-truth geometry, frame 4
GT
Predicted geometry, frame 4
Predicted

Errors on Held-Out Scenes

Depth vs. pose: Depth is the stronger output, with δ1 of 0.850 on validation and 0.809 on test, and the average error is only about 16%. Pose is weaker, and most of that weakness comes from translation. Mean rotation error is 12.2° on validation and 9.9° on test, while mean translation direction error is about 41° on both splits. Joint Pose AUC takes the larger of the two errors for each pair, so it is limited by translation and stays below 0.18 even at 30°. The possible reason for the weakness in translation is that the 8 frame came from a 32 frame window, so that camera barely move between some frames. This lead to a big change in direction for small mistakes.

Validation vs. test: Test results are slightly worse for depth (AbsRel 0.157 vs. 0.138) and for translation and joint AUC. Rotation stays close on both splits. Each split has only 28 scenes, so some of this gap may come from which scenes were selected.

Training behavior: Camera loss was still falling slowly late in training, and validation loss reached its minimum at epoch 843. The final epoch had a higher validation loss (0.852) and a worse AbsRel (0.151 vs. 0.138).

3.3Bonus: Point-Map Prediction

Question: Does predicting 3D points directly work better than unprojecting predicted depth with predicted cameras? Does adding a point loss also help the other tasks?

Settings: I added a point head with the same design as the depth head, plus a point loss with a weight of 1.0. Everything else matched Q3.2, including 1000 epochs and batch size 16. Both models were trained once and evaluated on the same 28 validation scenes.

Loss Curve

Point model training loss curve

Point Accuracy

MethodMean L2 (normalized)Mean L2 (m)Median L2 (m)Within 10 cm
Baseline, unprojected0.2720.5510.3458.7%
Point model, direct head0.2540.5180.3267.7%
Point model, unprojected0.2400.4880.30711.4%

The direct head was about 6% worse than unprojection from the same model. However, unprojection improved by 11.7% compared with the baseline, even though it does not use the head at test time. The best reconstruction therefore came from using the point loss during training and unprojecting at inference.

Error Sources

To find what limits unprojection, I replaced one prediction at a time with ground truth.

Unprojection inputs (mean L2, normalized)BaselinePoint modelChange
Predicted cameras, predicted depth0.2720.240-11.7%
Predicted cameras, GT depth0.2250.192-14.9%
GT cameras, predicted depth0.1380.134-3.2%

Cameras are the main limit. GT cameras reduced baseline error by 49%, while GT depth reduced it by only 17%. The point loss also seems to act mainly through the cameras, since the rows using predicted cameras improved much more than the row using GT cameras.

Effect on Other Tasks

Single checkpoints can be misleading, so this table averages validation metrics over the last 100 epochs.

MetricBaselinePoint model
Camera loss0.11440.1091
Depth loss0.2800.257
Depth AbsRel0.1490.142
Rotation error12.16°12.83°
Translation angle error41.95°42.22°
Pose AUC@300.1710.168

Camera and depth losses improved, but angular pose accuracy did not. Rotation looked better at the selected checkpoints (12.15° to 11.56°), but the late-training average does not support that.

Visual Comparison

Ground-truth point cloud
GT
Baseline unprojected point cloud
Baseline, unprojected
Point model direct head point cloud
Point model, direct head
Point model unprojected point cloud
Point model, unprojected

Discussion

The direct head was about 6% less accurate than unprojection from the same model. Unprojection builds in the camera geometry, while the head has to learn it implicitly for every pixel. The head is also a simple layer, so it may lack the capacity for precise points.

Training with the point loss still helped. Compared with the baseline, unprojected error dropped by about 12%, and camera and depth losses also improved. Angular pose accuracy did not improve, so the benefit may come from something those metrics ignore, such as translation scale, but I did not measure this.

Camera error was the main limit. Replacing predicted cameras with ground truth cut baseline error by 49%, while replacing predicted depth cut it by only 17%.

Limitations: Each model was trained with only one seed on a small dataset, and I only compared them on the validation set. The point loss weight was not tuned, and point error is still about 0.5 m for every method.

Conclusion: Direct point prediction did not beat unprojection, but the point loss was useful during training. The best approach was to train with the point head and still use unprojection for the final points. Better camera prediction would likely give the biggest improvement.