Assignment 4¶

Geethika Hemkumar (Andrew ID: ghemkuma)

1. Multi-View Data Preprocessing¶

Data Visualizations After Unprojection¶

data visualization after unprojection

Data Visualizations After Canonicalization¶

Screenshot¶

unprojected and canonicalized points

First Camera Center¶

points after canonicalization with first camera center coordinates

Valid-Point Centroid Distance¶

See valid-point centroid distance value in the lower right corner text area (value: 1.000 normalized units)

points after canonicalization with valid-point centroid distance

check_data.py Output¶

Expand for JSON Report From check_data.py
    {
        "split": "train",
        "dataset_samples": 224,
        "checked_samples": 4,
        "samples": [
            {
                "sequence_name": "e050c15a8d_frame_007080",
                "views": 8,
                "resolution": 224,
                "valid_depth_fraction": 0.9779127240180969,
                "max_reprojection_error_px": 0.00010844774078577757,
                "max_ray_reprojection_error_px": 9.392602805746719e-05,
                "centroid_distance": 1.0
            },
            {
                "sequence_name": "5f99900f09_frame_002820",
                "views": 8,
                "resolution": 224,
                "valid_depth_fraction": 0.9903116226196289,
                "max_reprojection_error_px": 0.00011108566104667261,
                "max_ray_reprojection_error_px": 0.00011108566104667261,
                "centroid_distance": 0.9999999403953552
            },
            {
                "sequence_name": "47b37eb6f9_frame_002720",
                "views": 8,
                "resolution": 224,
                "valid_depth_fraction": 0.9609947800636292,
                "max_reprojection_error_px": 8.508704195264727e-05,
                "max_ray_reprojection_error_px": 8.897326188161969e-05,
                "centroid_distance": 1.0
            },
            {
                "sequence_name": "4c5c60fa76_frame_006690",
                "views": 8,
                "resolution": 224,
                "valid_depth_fraction": 0.7516716122627258,
                "max_reprojection_error_px": 9.351845073979348e-05,
                "max_ray_reprojection_error_px": 9.423537994734943e-05,
                "centroid_distance": 1.0
            }
        ],
        "batch_shapes": {
            "rgb": [
                2,
                8,
                3,
                224,
                224
            ],
            "depth": [
                2,
                8,
                1,
                224,
                224
            ],
            "depth_valid": [
                2,
                8,
                1,
                224,
                224
            ],
            "K": [
                2,
                8,
                3,
                3
            ],
            "frame_numbers": [
                2,
                8
            ],
            "T_world_from_camera": [
                2,
                8,
                4,
                4
            ],
            "original_world_from_reference": [
                2,
                4,
                4
            ],
            "scene_scale": [
                2
            ],
            "points_world": [
                2,
                8,
                3,
                224,
                224
            ],
            "ray_origins": [
                2,
                8,
                3,
                224,
                224
            ],
            "ray_directions": [
                2,
                8,
                3,
                224,
                224
            ],
            "plucker": [
                2,
                8,
                6,
                224,
                224
            ]
        }
    }

Expand the above section for the check_data.py output. The reported shapes all make sense. For all scenes, depth validity is reasonable, point/ray reprojection errors are < 0.01, and centroid distances are approximately 1.

Explanation of Canonicalization, World-Space Ray Directions, and Plücker Moment¶

The process of canonicalization ensures that the geometry (camera poses, depth, and world-space points) are standardized across data samples. Canonicalization includes expressing all points in a common coordinate frame (the first camera's frame is used here) and normalizing the scene scale using the computed scene scale. This process reduces variation across samples and improves training stability.

World-space ray directions represent the directions of rays corresponding to each pixel in world coordinates. The directions of these rays are computed by unprojecting each pixel coordinate into the camera frame using the camera intrinsics $K$, then rotating them into world space using the rotation portion of $T\_world\_from\_camera$. The origin of each ray corresponding to the same camera will be the same, as all rays from the same camera view will originate from that camera's center.

Plücker moments are computed as the cross product of a given ray origin and direction. The Plücker moment is concatenated with the corresponding ray direction in order to form a six-dimensional representation of the ray. There are infinitely many points (origins) along a given ray, but the Plücker moment remains unchanged no matter which point along the ray is selected. Hence, this representation is unique to a ray in space, and further is preferred to using a concatenation of the ray direction and a selected origin.

2. Novel-View Synthesis with Mini-LVSM¶

2.1 Overfitting¶

Final Overfit Validation PSNR: 30.95

val_epoch000 for overfit test

2.2 Training and Evaluation¶

Loss Curves¶

train and validation loss curves for full LVSM training

Validation PSNR/SSIM for Each Ablation Setting¶

S Conditioning PSNR (dB) SSIM
1 correct 15.23 0.558
1 shuffled 11.80 0.487
1 none 12.46 0.529
2 correct 16.17 0.560
2 shuffled 11.46 0.472
2 none 12.44 0.526
4 correct 16.83 0.574
4 shuffled 11.60 0.476
4 none 12.47 0.537

Training Visualizations¶

Epoch 0¶

predictions from validation set after epoch 0

Epoch 50¶

predictions from validation set after epoch 50

Epoch 95¶

predictions from validation set after epoch 95

Validation Visualizations¶

S=1, Correct Target-Ray Conditioning¶

validation visualizations, S=1, conditioning=correct

S=1, Shuffled Target-Ray Conditioning¶

validation visualizations, S=1, conditioning=shuffled

S=1, No Target-Ray Conditioning¶

validation visualizations, S=1, conditioning=none

S=2, Correct Target-Ray Conditioning¶

validation visualizations, S=2, conditioning=correct

S=2, Shuffled Target-Ray Conditioning¶

validation visualizations, S=2, conditioning=shuffled

S=2, No Target-Ray Conditioning¶

validation visualizations, S=2, conditioning=none

S=4, Correct Target-Ray Conditioning¶

validation visualizations, S=4, conditioning=correct

S=4, Shuffled Target-Ray Conditioning¶

validation visualizations, S=4, conditioning=shuffled

S=4, No Target-Ray Conditioning¶

validation visualizations, S=4, conditioning=none

Training Settings¶

Used the default settings:

Setting Value
Epochs 100
Learning rate 2e-4
Weight decay 0.05
Betas (0.9, 0.95)
Warmup steps 200
Grad clip 1.0
AMP true

Effect of Number of Source Views and Ray-Conditioning Ablations¶

Increasing the number of source views improves the PSNR/SSIM when the correct (true target rays) target-ray conditioning is used. PSNR/SSIM scores do not vary meaningfully with the number of source views within the shuffled or none target-ray conditioning settings.

Qualitatively, the target view predictions with the none target-ray conditioning setting are near 0 across pixels. The shuffled setting produces realistic target-view predictions, but as expected they do not match the ground truth target view as the target ray that prediction was conditioned on does not match the target view that is being predicted. Target view predictions with correct target-view conditioning are most accurate, as the target ray used for prediction conditioning matches the target view being predicted.

3. Multi-View Geometry with Mini-VGGT¶

3.1 Overfitting¶

Loss Curve¶

training loss curve for VGGT overfit test

Ground-Truth (GT) and Predicted Geometry Comparisons¶

Full Geometry¶

GT

q31 ground-truth, full geometry

Predicted

q31 predicted, full geometry

View 0¶

GT

q31 ground-truth, view 0

Predicted

q31 predicted, view 0

View 3¶

GT

q31 ground-truth, view 3

Predicted

q31 predicted, view 3

View 7¶

GT

q31 ground-truth, view 7

Predicted

q31 predicted, view 7

3.2 Training and Evaluation¶

Training/Validation Loss Curves¶

training and validation loss curves for full VGGT training

Validation Set Metrics¶

Metric Value
Depth AbsRel 0.152
Depth δ1 0.811
Rotation AUC@5° / @15° / @30° 0.194 / 0.520 / 0.707
Translation AUC@5° / @15° / @30° 0.012 / 0.074 / 0.188
Joint Pose AUC@5° / @15° / @30° 0.005 / 0.056 / 0.165

Test Set Metrics¶

Metric Value
Depth AbsRel 0.156
Depth δ1 0.799
Rotation AUC@5° / @15° / @30° 0.192 / 0.528 / 0.721
Translation AUC@5° / @15° / @30° 0.010 / 0.056 / 0.161
Joint Pose AUC@5° / @15° / @30° 0.001 / 0.040 / 0.147

Ground-Truth (GT) and Predicted Geometry Comparisons¶

Scene 1¶

GT

q32 ground-truth, scene 1

Predicted

q32 predicted, scene 1

Scene 2¶

GT

q32 ground-truth, scene 2

Predicted

q32 predicted, scene 2

Scene 3¶

GT

q32 ground-truth, scene 3

Predicted

q32 predicted, scene 3

Scene 4¶

GT

q32 ground-truth, scene 4

Predicted

q32 predicted, scene 4

Training Settings¶

Used the default settings:

Setting Value
Epochs 1000
Learning rate 6e-4
Weight decay 0.01
Min LR ratio 0.05
Warmup epochs 10

Explanation of Errors on Held-Out Scenes¶

Across all four held-out scenes visualized above, the predicted and ground truth depth/point clouds have similar global structures. The predicted depth maps appear "blurry" compared to the ground truth depth maps, meaning that the boundaries (i.e., depth discontinuities) between objects in the scene and the environment (floor, walls, etc) are less distinct. A similar idea holds for the predicted point clouds/camera poses: the general shapes of the scene / objects are distinguishable, but the structures aren't as well-defined and detailed compared to those in the corresponding ground truth scenes.