Assignment 4: Multi-View Transformers
1. Multi-View Data Preprocessing
1.2Depth Unprojection
Data visualization after unprojection:

1.3Camera and Depth Canonicalization
Data visualization after canonicalization:

Canonicalization: A scene's coordinates depend on an arbitrary origin and scale, so raw values vary widely between scenes. Canonicalization describes every scene from the same starting point at the same size. It take camera 0 as the reference and compute every pose relative to it:
The equation above make an identity matrix. After that, the average position for all valid 3D points, which is the scene's centroid, is obtained. The scene scale is the distance from camera 0 to the centroid of the valid points, computed in this new frame. Camera translations, depth and points are all divided by , so the centroid ends up at distance 1 from the origin. Rotations do not change.
1.4Camera Rays
World-space ray directions: Each pixel in the images correspond to a ray leaving the camera center, multiplying its homogeneous coordinates by the inverse intrinsics of the camera gives the direction in camera space:
Rotating by the camera-to-world rotation matrix and normalizing it yield the unit world-space direction:
Only the rotation is applied because a direction has no position. The translation gives the ray origin instead.
1.5Plücker Coordinates
Plücker Moment: If explaining the ray solely by its direction , it only gives the orientation of the ray so parallel rays will be identical. To solve this, a Plücker Moment is added to describe the difference:
Here, is the ray origin, is perpendicular to the plane containing the ray and the origin, and since is a unit vector, is the distance from the origin to the ray's line.
1.7Final Check
check_data.py output:
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00010235906665911898,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM
2.1Overfitting

val_epoch000.png2.2Training and Evaluation
Training Settings
| Setting | Value |
|---|---|
| Batch size | 4 |
| Eval frames | 32 |
| Patch size | 8 |
| d_model | 384 |
| Layers | 8 |
| Head dim | 64 |
| MLP ratio | 4.0 |
| QK norm | true |
| Epochs | 100 |
| Learning rate | 2e-4 |
| Weight decay | 0.05 |
| Betas | (0.9, 0.95) |
| Warmup steps | 200 |
| Grad clip | 1.0 |
| AMP | true |
| Source views | 2 |
Training curves

Validation metrics
The table below used PSNR(dB)/SSIM format
| Source views | Correct | Shuffled | None |
|---|---|---|---|
| 1 | 15.20 / 0.558 | 11.80 / 0.487 | 12.48 / 0.528 |
| 2 | 16.13 / 0.560 | 11.49 / 0.472 | 12.46 / 0.527 |
| 4 | 16.87 / 0.575 | 11.59 / 0.473 | 12.48 / 0.536 |
Target images evaluated: 196 for S=1, 168 for S=2, 112 for S=4.
Training visualizations



Evaluation visualizations









Effect of the number of source views: When the model gets the correct rays, adding source views helps steadily. PSNR climbs from 15.20 dB with 1 view to 16.87 dB with 4, and SSIM goes from 0.558 to 0.575. This is because each extra view shows more of the scene, so the model has less to predict on its own.
Effect of ray conditioning: The effect of rays is huge. If the ray is taken away, PSNR falls to roughly 11.5–12.5 dB no matter how many source views we provide, so the benefit of extra views vanishes too. Without rays, the model has no way of knowing which viewpoint we're asking for, so it hedges and produces nearly the same blurry, averaged image every time. Shuffled rays are even worse than no rays. The model confidently renders a view for the wrong pose, which under MSE a this costs more than a vague average.
3. Multi-View Geometry with Mini-VGGT
3.1Overfitting
Loss Curve



3.2Training and Evaluation
Training Settings
| Setting | Value |
|---|---|
| Views / sampling range | 8 / 32 |
| Image size | 224 |
| Batch size | 16 (14 steps per epoch) |
| Encoder | DINOv2 ViT-B/14 (frozen) |
| Embedding dim / heads / layers | 384 / 6 / 8 |
| Point head | Off |
| Epochs | 1000 |
| Optimizer | AdamW |
| Learning rate | 6e-4, decaying to 3e-5 |
| Warmup epochs | 10 |
| Weight decay | 0.01 |
| Grad clip | 1.0 |
| Loss weights | Camera 5.0, depth 1.0 |
| Precision | bf16 |
| GPUs | 1 |
Loss Curve

Metrics
| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.138 | 0.157 |
| Depth δ1 | 0.850 | 0.809 |
| Rotation AUC @ 5° | 0.208 | 0.178 |
| Rotation AUC @ 15° | 0.522 | 0.505 |
| Rotation AUC @ 30° | 0.702 | 0.712 |
| Translation AUC @ 5° | 0.018 | 0.013 |
| Translation AUC @ 15° | 0.096 | 0.058 |
| Translation AUC @ 30° | 0.205 | 0.159 |
| Joint Pose AUC @ 5° | 0.007 | 0.004 |
| Joint Pose AUC @ 15° | 0.070 | 0.036 |
| Joint Pose AUC @ 30° | 0.178 | 0.141 |
GT vs. Predicted Geometry








Errors on Held-Out Scenes
Depth vs. pose: Depth is the stronger output, with δ1 of 0.850 on validation and 0.809 on test, and the average error is only about 16%. Pose is weaker, and most of that weakness comes from translation. Mean rotation error is 12.2° on validation and 9.9° on test, while mean translation direction error is about 41° on both splits. Joint Pose AUC takes the larger of the two errors for each pair, so it is limited by translation and stays below 0.18 even at 30°. The possible reason for the weakness in translation is that the 8 frame came from a 32 frame window, so that camera barely move between some frames. This lead to a big change in direction for small mistakes.
Validation vs. test: Test results are slightly worse for depth (AbsRel 0.157 vs. 0.138) and for translation and joint AUC. Rotation stays close on both splits. Each split has only 28 scenes, so some of this gap may come from which scenes were selected.
Training behavior: Camera loss was still falling slowly late in training, and validation loss reached its minimum at epoch 843. The final epoch had a higher validation loss (0.852) and a worse AbsRel (0.151 vs. 0.138).
3.3Bonus: Point-Map Prediction
Question: Does predicting 3D points directly work better than unprojecting predicted depth with predicted cameras? Does adding a point loss also help the other tasks?
Settings: I added a point head with the same design as the depth head, plus a point loss with a weight of 1.0. Everything else matched Q3.2, including 1000 epochs and batch size 16. Both models were trained once and evaluated on the same 28 validation scenes.
Loss Curve

Point Accuracy
| Method | Mean L2 (normalized) | Mean L2 (m) | Median L2 (m) | Within 10 cm |
|---|---|---|---|---|
| Baseline, unprojected | 0.272 | 0.551 | 0.345 | 8.7% |
| Point model, direct head | 0.254 | 0.518 | 0.326 | 7.7% |
| Point model, unprojected | 0.240 | 0.488 | 0.307 | 11.4% |
The direct head was about 6% worse than unprojection from the same model. However, unprojection improved by 11.7% compared with the baseline, even though it does not use the head at test time. The best reconstruction therefore came from using the point loss during training and unprojecting at inference.
Error Sources
To find what limits unprojection, I replaced one prediction at a time with ground truth.
| Unprojection inputs (mean L2, normalized) | Baseline | Point model | Change |
|---|---|---|---|
| Predicted cameras, predicted depth | 0.272 | 0.240 | -11.7% |
| Predicted cameras, GT depth | 0.225 | 0.192 | -14.9% |
| GT cameras, predicted depth | 0.138 | 0.134 | -3.2% |
Cameras are the main limit. GT cameras reduced baseline error by 49%, while GT depth reduced it by only 17%. The point loss also seems to act mainly through the cameras, since the rows using predicted cameras improved much more than the row using GT cameras.
Effect on Other Tasks
Single checkpoints can be misleading, so this table averages validation metrics over the last 100 epochs.
| Metric | Baseline | Point model |
|---|---|---|
| Camera loss | 0.1144 | 0.1091 |
| Depth loss | 0.280 | 0.257 |
| Depth AbsRel | 0.149 | 0.142 |
| Rotation error | 12.16° | 12.83° |
| Translation angle error | 41.95° | 42.22° |
| Pose AUC@30 | 0.171 | 0.168 |
Camera and depth losses improved, but angular pose accuracy did not. Rotation looked better at the selected checkpoints (12.15° to 11.56°), but the late-training average does not support that.
Visual Comparison




Discussion
The direct head was about 6% less accurate than unprojection from the same model. Unprojection builds in the camera geometry, while the head has to learn it implicitly for every pixel. The head is also a simple layer, so it may lack the capacity for precise points.
Training with the point loss still helped. Compared with the baseline, unprojected error dropped by about 12%, and camera and depth losses also improved. Angular pose accuracy did not improve, so the benefit may come from something those metrics ignore, such as translation scale, but I did not measure this.
Camera error was the main limit. Replacing predicted cameras with ground truth cut baseline error by 49%, while replacing predicted depth cut it by only 17%.
Limitations: Each model was trained with only one seed on a small dataset, and I only compared them on the validation set. The point loss weight was not tuned, and point error is still about 0.5 m for every method.
Conclusion: Direct point prediction did not beat unprojection, but the point loss was useful during training. The best approach was to train with the point head and still use unprojection for the final points. Better camera prediction would likely give the biggest improvement.