Assignment 4: Multi-View Transformers
1. Multi-View Data Preprocessing
Unprojection visualization:
| Front view |
 |
| Side view |
 |
| Top-down view |
 |
Canonicalization visualization:
| Front view |
 |
| Side view |
 |
| Top-down view |
 |
First camera center (X, Y, Z): (0.0000, -0.0000, -0.0000) normalized units.
Scene scale: 1.867 m.
Valid-point centroid distance: 1.000 normalized units.
check_data.py output (expand for json output):
Details
``` json
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.0001102154201362282,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.5367431640625e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010647465387592092,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 0.9999999403953552
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
```
Canonicalization: The process of aligning 3D object data and camera poses into a default standardized format, adjusting scale, translation and pose. In this case, the scale and translation of the scene is adjusted such that the valid-point centroid lies at origin and the camera is a distance of 1 unit from the origin.
World-space ray directions: A 3D unit vector that describes the orientation of a camera ray with respect to the world coordinate system as opposed to the local camera coordinate system.
Plucker moment: This represents a continuous line in 3D space without specifying a starting point. The plucker moment describes the cross product of a point on the line with the direction vector of the line.
2. Novel-View Synthesis with Mini-LVSM
2.1 Overfitting
Final overfitting PSNR:
training: 29.057 dB
validation: 36.616 dB
Visualization

2.2 Training and Evaluation
Training loss:

PSNR ablation:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
15.204 |
11.752 |
12.445 |
| 2 source views |
16.121 |
11.435 |
12.428 |
| 4 source views |
16.869 |
11.533 |
12.474 |
SSIM ablation:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
0.554 |
0.483 |
0.525 |
| 2 source views |
0.556 |
0.468 |
0.522 |
| 4 source views |
0.572 |
0.469 |
0.533 |
Evaluation visualization with ablations:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
 |
 |
 |
| 2 source view |
 |
 |
 |
| 4 source view |
 |
 |
 |
Training visualization gif:

Training settings (expand dropdown for json config):
batch_size was changed to 4 to fit into VRAM.
Details
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"batch_size": 4,
"eval_num_frames": 32
},
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true,
"vis_every": 5,
"output_dir": "outputs/lvsm"
},
"num_source_views": 2
}
Increasing the number of source views from 1 to 2 yields the highest improvement in output while increasing it from 2 to 4 shows diminishing returns.
Changing the ray conditioning from correct to shuffled reduces the quality of the output drastically, which indicates that the model is using the Plucker rays as part of its attention mechanism. However changing it to none removes all directional context, rendering a blurry average output that is slightly better metrically, indicating a learned prior.
2.3 Analysis and exploration
Question: How does changing the training view count from 2 to 4 affect results and quality? How does this change the evaluation with 1, 2 and 4 source views?
PSNR ablation:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
15.184 |
11.693 |
12.379 |
| 2 source views |
16.297 |
11.329 |
12.346 |
| 4 source views |
17.280 |
11.362 |
12.383 |
SSIM ablation:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
0.552 |
0.477 |
0.523 |
| 2 source views |
0.556 |
0.460 |
0.520 |
| 4 source views |
0.573 |
0.456 |
0.529 |
Training visualization gif:

Evaluation visualization with ablations:
| Ray-conditioning: |
Correct |
Shuffled |
None |
| 1 source view |
 |
 |
 |
| 2 source view |
 |
 |
 |
| 4 source view |
 |
 |
 |
Training settings (expand dropdown for json config):
num_source_views was changed from 2 to 4.
Details
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"batch_size": 4,
"eval_num_frames": 32
},
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true,
"vis_every": 5,
"output_dir": "outputs/lvsm"
},
"num_source_views":4
}
Conclusion from experiment
Visually, compared to using 2 source views, the outputs from the model trained with 4 views produces visually slightly better outputs however the difference is slight, as demonstrated by the PSNR and SSIM score improvements as well.
This improvement seems to produce diminishing returns possibly because the jump from 1 to 2 contributes to parallax and 3D point triangulation while the jump from 2 to 4 adds mostly overlapping visual data.
3. Multi-View Geometry with Mini-VGGT
3.1 Overfitting
Loss curves:

GT vs Prediction
| Front view (GT) |
Front view (Prediction) |
 |
.png) |
| Side view (GT) |
Side view (Prediction) |
.png) |
.png) |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
3.2. Training and Evaluation
Loss curves:

GT vs Prediction: Scene 1
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
GT vs Prediction: Scene 2
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
GT vs Prediction: Scene 3
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
GT vs Prediction: Scene 4
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
Validation metrics
| Metric |
Value |
| Depth AbsRel |
0.146 |
| Depth δ1 |
0.836 |
| Rotation AUC 5 deg |
0.168 |
| Rotation AUC 15 deg |
0.472 |
| Rotation AUC 30 deg |
0.657 |
| Translation AUC 5 deg |
0.009 |
| Translation AUC 15 deg |
0.066 |
| Translation AUC 30 deg |
0.170 |
| Joint Pose AUC 5 deg |
0.003 |
| Joint Pose AUC 15 deg |
0.051 |
| Joint Pose AUC 30 deg |
0.146 |
Test metrics
| Metric |
Value |
| Depth AbsRel |
0.162 |
| Depth δ1 |
0.804 |
| Rotation AUC 5 deg |
0.182 |
| Rotation AUC 15 deg |
0.496 |
| Rotation AUC 30 deg |
0.691 |
| Translation AUC 5 deg |
0.012 |
| Translation AUC 15 deg |
0.052 |
| Translation AUC 30 deg |
0.143 |
| Joint Pose AUC 5 deg |
0.003 |
| Joint Pose AUC 15 deg |
0.039 |
| Joint Pose AUC 30 deg |
0.132 |
Training settings (expand dropdown for json config):
batch_size changed to 10 and num_workers set to 10 to fit in VRAM:
Details
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 10,
"num_workers": 4
},
"model": {
"embed_dim": 384,
"num_heads": 6,
"num_layers": 8,
"predict_points": false,
"encoder_name": "dinov2_vitb14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 1000,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"output_dir": "outputs/vggt",
"overfit_scenes": 1,
"overfit_steps_per_epoch": 25,
"min_lr_ratio": 0.05,
"log_every": 10,
"warmup_epochs": 10,
"overfit_epochs": 40,
"overfit_learning_rate": 0.001
}
}
The depth values of the points for held-out scenes are inaccurate possibly due to incorrect scale outputs.
3.3: Analysis and Exploration
Question: How does training the DINO encoder affect performance compared with keeping it frozen?
Experimental settings: Same as above but DINO gradients are enabled during training instead of frozen.
Loss curves:

GT vs Prediction: Scene 1
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
GT vs Prediction: Scene 2
| Front view (GT) |
Front view (Prediction) |
 |
 |
| Side view (GT) |
Side view (Prediction) |
 |
 |
| Top-down view (GT) |
Top-down view (Prediction) |
 |
 |
Validation metrics
| Metric |
Value with unfrozen DINO backbone |
Baseline value |
Better |
| Depth AbsRel |
0.237 |
0.146 |
Lower |
| Depth δ1 |
0.658 |
0.836 |
Higher |
| Rotation AUC 5 deg |
0.115 |
0.168 |
Higher |
| Rotation AUC 15 deg |
0.353 |
0.472 |
Higher |
| Rotation AUC 30 deg |
0.552 |
0.657 |
Higher |
| Translation AUC 5 deg |
0.002 |
0.009 |
Higher |
| Translation AUC 15 deg |
0.025 |
0.066 |
Higher |
| Translation AUC 30 deg |
0.092 |
0.170 |
Higher |
| Joint Pose AUC 5 deg |
0.0005 |
0.003 |
Higher |
| Joint Pose AUC 15 deg |
0.014 |
0.051 |
Higher |
| Joint Pose AUC 30 deg |
0.071 |
0.146 |
Higher |
Test metrics
| Metric |
Value with unfrozen DINO backbone |
Baseline value |
Better |
| Depth AbsRel |
0.273 |
0.162 |
Lower |
| Depth δ1 |
0.604 |
0.804 |
Higher |
| Rotation AUC 5 deg |
0.140 |
0.182 |
Higher |
| Rotation AUC 15 deg |
0.376 |
0.496 |
Higher |
| Rotation AUC 30 deg |
0.577 |
0.691 |
Higher |
| Translation AUC 5 deg |
0.003 |
0.012 |
Higher |
| Translation AUC 15 deg |
0.042 |
0.052 |
Higher |
| Translation AUC 30 deg |
0.124 |
0.143 |
Higher |
| Joint Pose AUC 5 deg |
0.001 |
0.003 |
Higher |
| Joint Pose AUC 15 deg |
0.039 |
0.039 |
Higher |
| Joint Pose AUC 30 deg |
0.132 |
0.132 |
Higher |
Conclusion from experiment
Visually, unfreezing the DINO backbone produces worse results, with depth errors more obviously seen.
This degradation is because the half the model is trained (the DINO backbone) while the other half (the VGGT heads) are untrained. This causes gradients to affect the trained backbone, disturbing the trained gradients. This effect is further amplified by the fact that both the frozen and unfrozen models are trained for the same number of training steps; the frozen model has way fewer parameters to optimize compared to the unfrozen model.