Assignment 4¶
Geethika Hemkumar (Andrew ID: ghemkuma)
1. Multi-View Data Preprocessing¶
Data Visualizations After Unprojection¶

check_data.py Output¶
Expand for JSON Report From check_data.py
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00010844774078577757,
"max_ray_reprojection_error_px": 9.392602805746719e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00011108566104667261,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 8.508704195264727e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 9.351845073979348e-05,
"max_ray_reprojection_error_px": 9.423537994734943e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Expand the above section for the check_data.py output. The reported shapes all make sense. For all scenes, depth validity is reasonable, point/ray reprojection errors are < 0.01, and centroid distances are approximately 1.
Explanation of Canonicalization, World-Space Ray Directions, and Plücker Moment¶
The process of canonicalization ensures that the geometry (camera poses, depth, and world-space points) are standardized across data samples. Canonicalization includes expressing all points in a common coordinate frame (the first camera's frame is used here) and normalizing the scene scale using the computed scene scale. This process reduces variation across samples and improves training stability.
World-space ray directions represent the directions of rays corresponding to each pixel in world coordinates. The directions of these rays are computed by unprojecting each pixel coordinate into the camera frame using the camera intrinsics $K$, then rotating them into world space using the rotation portion of $T\_world\_from\_camera$. The origin of each ray corresponding to the same camera will be the same, as all rays from the same camera view will originate from that camera's center.
Plücker moments are computed as the cross product of a given ray origin and direction. The Plücker moment is concatenated with the corresponding ray direction in order to form a six-dimensional representation of the ray. There are infinitely many points (origins) along a given ray, but the Plücker moment remains unchanged no matter which point along the ray is selected. Hence, this representation is unique to a ray in space, and further is preferred to using a concatenation of the ray direction and a selected origin.
2. Novel-View Synthesis with Mini-LVSM¶
2.2 Training and Evaluation¶
Loss Curves¶

Validation PSNR/SSIM for Each Ablation Setting¶
| S | Conditioning | PSNR (dB) | SSIM |
|---|---|---|---|
| 1 | correct | 15.23 | 0.558 |
| 1 | shuffled | 11.80 | 0.487 |
| 1 | none | 12.46 | 0.529 |
| 2 | correct | 16.17 | 0.560 |
| 2 | shuffled | 11.46 | 0.472 |
| 2 | none | 12.44 | 0.526 |
| 4 | correct | 16.83 | 0.574 |
| 4 | shuffled | 11.60 | 0.476 |
| 4 | none | 12.47 | 0.537 |
Training Visualizations¶
Epoch 0¶

Epoch 50¶

Epoch 95¶

Validation Visualizations¶
S=1, Correct Target-Ray Conditioning¶

S=1, Shuffled Target-Ray Conditioning¶

S=1, No Target-Ray Conditioning¶

S=2, Correct Target-Ray Conditioning¶

S=2, Shuffled Target-Ray Conditioning¶

S=2, No Target-Ray Conditioning¶

S=4, Correct Target-Ray Conditioning¶

S=4, Shuffled Target-Ray Conditioning¶

S=4, No Target-Ray Conditioning¶

Training Settings¶
Used the default settings:
| Setting | Value |
|---|---|
| Epochs | 100 |
| Learning rate | 2e-4 |
| Weight decay | 0.05 |
| Betas | (0.9, 0.95) |
| Warmup steps | 200 |
| Grad clip | 1.0 |
| AMP | true |
Effect of Number of Source Views and Ray-Conditioning Ablations¶
Increasing the number of source views improves the PSNR/SSIM when the correct (true target rays) target-ray conditioning is used. PSNR/SSIM scores do not vary meaningfully with the number of source views within the shuffled or none target-ray conditioning settings.
Qualitatively, the target view predictions with the none target-ray conditioning setting are near 0 across pixels. The shuffled setting produces realistic target-view predictions, but as expected they do not match the ground truth target view as the target ray that prediction was conditioned on does not match the target view that is being predicted. Target view predictions with correct target-view conditioning are most accurate, as the target ray used for prediction conditioning matches the target view being predicted.
3. Multi-View Geometry with Mini-VGGT¶
3.2 Training and Evaluation¶
Training/Validation Loss Curves¶

Validation Set Metrics¶
| Metric | Value |
|---|---|
| Depth AbsRel | 0.152 |
| Depth δ1 | 0.811 |
| Rotation AUC@5° / @15° / @30° | 0.194 / 0.520 / 0.707 |
| Translation AUC@5° / @15° / @30° | 0.012 / 0.074 / 0.188 |
| Joint Pose AUC@5° / @15° / @30° | 0.005 / 0.056 / 0.165 |
Test Set Metrics¶
| Metric | Value |
|---|---|
| Depth AbsRel | 0.156 |
| Depth δ1 | 0.799 |
| Rotation AUC@5° / @15° / @30° | 0.192 / 0.528 / 0.721 |
| Translation AUC@5° / @15° / @30° | 0.010 / 0.056 / 0.161 |
| Joint Pose AUC@5° / @15° / @30° | 0.001 / 0.040 / 0.147 |
Ground-Truth (GT) and Predicted Geometry Comparisons¶
Scene 1¶
GT

Predicted

Scene 2¶
GT

Predicted

Scene 3¶
GT

Predicted

Scene 4¶
GT

Predicted

Training Settings¶
Used the default settings:
| Setting | Value |
|---|---|
| Epochs | 1000 |
| Learning rate | 6e-4 |
| Weight decay | 0.01 |
| Min LR ratio | 0.05 |
| Warmup epochs | 10 |
Explanation of Errors on Held-Out Scenes¶
Across all four held-out scenes visualized above, the predicted and ground truth depth/point clouds have similar global structures. The predicted depth maps appear "blurry" compared to the ground truth depth maps, meaning that the boundaries (i.e., depth discontinuities) between objects in the scene and the environment (floor, walls, etc) are less distinct. A similar idea holds for the predicted point clouds/camera poses: the general shapes of the scene / objects are distinguishable, but the structures aren't as well-defined and detailed compared to those in the corresponding ground truth scenes.












