Name: Gangadhar Nageswar
Andrew ID: vnageswa
Late Days Used: 2

Question 1 : 8-point and 7-point algorithm

A1. F matrix using 8-point algorithm

Description of GIF

Fundamental Matrix:
[[-5.74746359e-07, 1.52344835e-07, -1.88821338e-03],
[ 1.50407127e-06, 6.87859250e-07, 6.76008549e-03],
[ 3.18192544e-03, -7.56595972e-03, -7.99361620e-01]]

Essential Matrix:
[[-0.81269241, 0.21541588, -2.35504428],
[ 2.12675954, 0.97263424, 9.09084225],
[ 4.65830811, -8.48677963, -0.32124688]]

Description of GIF

Fundamental Matrix:
[[-1.46486423e-08, -3.11057196e-07, 2.54961985e-05],
[-4.86952282e-08, 2.78927493e-08, 3.05667982e-03],
[-1.31286915e-04, -2.82557971e-03, 1.22823460e-01]]

Essential Matrix:
[[-0.02275717, -0.48323809, -0.11786026],
[-0.07564972, 0.04333235, 3.78604024],
[-0.19668836, -3.75270467, 0.0548045]]

Explanation of the implemnentation:
Procedure:
1. Normalize the coordinates of points from both images to ensure better numerical stability, obtaining transformations `t1` and `t2` for points `pts1` and `pts2`, respectively.
2. Construct matrix `A` by using the epipolar constraint equation, where for each pair of corresponding points `p1 = [u1, v1, w1]^T` and `p2 = [u2, v2, w2]^T`, the row becomes `[u2*u1, u2*v1, u2*w1, v2*u1, v2*v1, v2*w1, w2*u1, w2*v1, w2*w1]`.
3. Solve for the initial estimate of the fundamental matrix `F` by applying Singular Value Decomposition (SVD) on matrix `A` and selecting the vector corresponding to the smallest singular value of `A` (last row of `Vh`).
4. Enforce the rank-2 constraint on `F` by applying SVD on `F` and setting the smallest singular value to zero, ensuring that `F` has rank 2.
5. Denormalize `F` to the original coordinate system by using the transformations `t1` and `t2` from the normalization step, resulting in `F = t2.T * F_norm * t1`.
6. Return the resulting fundamental matrix `F` which satisfies the epipolar constraint between the corresponding points in the two images.

Key equations:
1. Epipolar constraint: `p2.T * F * p1 = 0`.
2. Normalize points to ensure zero mean and unit standard deviation for better accuracy and numerical stability.
3. Rank-2 constraint: Set the smallest singular value of `F` to zero to ensure it has rank 2.
4. Denormalization of F: `F = t2.T * F_norm * t1`.

A2. Esential Matrix using 8-point algorithm

Essential Matrix:
[[-0.81269241, 0.21541588, -2.35504428],
[ 2.12675954, 0.97263424, 9.09084225],
[ 4.65830811, -8.48677963, -0.32124688]]

Essential Matrix:
[[-0.02275717, -0.48323809, -0.11786026],
[-0.07564972, 0.04333235, 3.78604024],
[-0.19668836, -3.75270467, 0.0548045]]

Explanation of the implemnentation:
Procedure:
1. Start with the fundamental matrix `F` computed using the 8-point algorithm or another method.
2. Obtain the intrinsic matrices `K1` and `K2` for the two cameras, representing the internal calibration parameters.
3. Compute the essential matrix `E` by transforming `F` with the intrinsic matrices using the equation `E = K2.T @ F @ K1`.
4. The resulting essential matrix `E` encapsulates the relative rotation and translation between the two camera views under the epipolar geometry.

Key equation:
1. Essential matrix: `E = K2.T * F * K1`.

B. 7-pt algorithm

Description of GIF

Fundamental Matrix:
[[-5.74746359e-07, 1.52344835e-07, -1.88821338e-03],
[ 1.50407127e-06, 6.87859250e-07, 6.76008549e-03],
[ 3.18192544e-03, -7.56595972e-03, -7.99361620e-01]]

Essential Matrix:
[[-0.81269241, 0.21541588, -2.35504428],
[ 2.12675954, 0.97263424, 9.09084225],
[ 4.65830811, -8.48677963, -0.32124688]]

Description of GIF

Fundamental Matrix:
[[-3.68946642e-10, -3.21013410e-07, -7.43107559e-04],
[ 3.87229606e-07, -9.73613258e-08, -1.65210131e-03],
[ 1.00131754e-03, 1.63783671e-03, 1.10107670e-01]]

Essential Matrix:
[[-1.42438039e-03, -1.23932611e+00, -2.05621920e+00],
[ 1.49496484e+00, -3.75879726e-01, -3.02279474e+00],
[ 2.68490225e+00, 2.70219654e+00, 1.80760301e-01]]

Explanation of the implemnentation:
Procedure:
1. Normalize the coordinates of points from both images to enhance numerical stability, obtaining transformations `t1` and `t2` for points `pts1` and `pts2`, respectively.
2. Construct matrix `A` for each pair of corresponding points `p1 = [u1, v1, w1]^T` and `p2 = [u2, v2, w2]^T` by creating rows `[u2*u1, u2*v1, u2*w1, v2*u1, v2*v1, v2*w1, w2*u1, w2*v1, w2*w1]`.
3. Perform Singular Value Decomposition (SVD) on `A` to get the two potential fundamental matrices `F1` and `F2` corresponding to the last two rows of `Vh`.
4. Define a range of `alpha` values to linearly combine `F1` and `F2` as `F(alpha) = alpha * F1 + (1 - alpha) * F2`.
5. Evaluate the determinant of `F(alpha)` at each `alpha` and fit a polynomial to determine where the determinant is zero, obtaining a cubic polynomial with roots.
6. Identify the real roots from the polynomial, selecting a valid `alpha` value to compute the final fundamental matrix `F`.
7. Denormalize `F` to the original coordinate system using transformations `t1` and `t2`, resulting in `F = t2.T * F * t1`.
8. Return the resulting fundamental matrix `F` which meets the epipolar constraint between the points in the two images.

Key equations:
1. Linear combination of potential fundamental matrices: `F(alpha) = alpha * F1 + (1 - alpha) * F2`.
2. Polynomial fitting for determinant: `det(F(alpha)) = 0`.
3. Denormalization of F: `F = t2.T * F * t1`.

Question 2 - RANSAC

Output 1 - Bench

RANSAC with 8-point algorithm
Description of GIF
RANSAC with 7-point algorithm
Description of GIF
Fraction of inliers
Description of GIF

Output 2 - Hydrant

RANSAC with 8-point algorithm
Description of GIF
RANSAC with 7-point algorithm
Description of GIF
Fraction of inliers
Description of GIF

Output 3 - Ball

RANSAC with 8-point algorithm
Description of GIF
RANSAC with 7-point algorithm
Description of GIF
Fraction of inliers
Description of GIF

Implementation details:
Procedure:
1. Initialize the RANSAC process to estimate the best fundamental matrix `F` that maximizes inliers. Set parameters such as number of iterations (`num_itrs`), inlier threshold (`eps`), and algorithm type (`7-pt` or `8-pt`).
2. For each iteration, randomly select a subset of points (7 for `7-pt` or 8 for `8-pt` algorithm).
3. Compute a candidate fundamental matrix `F` from the subset using the specified algorithm.
4. Calculate the epipolar distances between all points in `pts1` and `pts2` using `F`.
5. Mark points within distance `eps` as inliers. Count the inliers for this candidate `F`.
6. If the current `F` has more inliers than previous ones, update `F_best`, store the inlier points, and record the fraction of inliers.
7. Repeat steps 2-6 for the specified number of iterations (`num_itrs`) to maximize inliers.
Key Criteria and Outputs:
1. Inliers: Points with epipolar distances below threshold `eps` are considered inliers.
2. Criteria: The euclidean distance between the epipolar lines and the corresponding points should be less than the threshold `eps`.
3. Best Solution: The final fundamental matrix `F_best` is the one with the most inliers.

Question 3 - Triangulation

Reconstructed Image
Description of GIF

Triangulation implemnentation details and key equations.

Procedure:
1. Initialize empty lists `X` for 3D points and `point_rgb` for colors. Convert `pts1` and `pts2` to `float32` type.
2. For each point pair in `pts1` and `pts2`, construct a matrix `A` using projection matrices `P1` and `P2`.
3. Define `A` as: `A = [[-pts1[i,0] * P1[2,:] + P1[0,:]], [pts1[i,1] * P1[2,:] - P1[1,:]], [-pts2[i,0] * P2[2,:] + P2[0,:]], [pts2[i,1] * P2[2,:] - P2[1,:]]]`.
4. Perform SVD on `A` to get `Xi`, the homogeneous 3D coordinates. Use the last row of `Vh` for `Xi`.
5. Normalize `Xi` by dividing by its last element and add it to `X`.
6. Retrieve the color of each point in `pts1` from `img1` and store it in `point_rgb` (convert from BGR to RGB).
7. Convert `X` and `point_rgb` to numpy arrays, normalizing `point_rgb` to range [0, 1].
8. Return the arrays `X` and `point_rgb`.

Key Equations:
1. Triangulation equation: Construct matrix `A` so that `A * Xi = 0`, where `Xi` is the 3D point in homogeneous coordinates.
2. System for `A`: `A = [[-x * P1[2,:] + P1[0,:]], [y * P1[2,:] - P1[1,:]], [-x' * P2[2,:] + P2[0,:]], [y' * P2[2,:] - P2[1,:]]]`.
3. SVD Solution: Compute `Xi` as the smallest singular vector of `A` and normalize: `Xi = Xi / Xi[-1]`.

Question 4 - COLMAP

Output 1 - Cart

Input Images
Description of GIF
COLMAP Reconstruction
Description of GIF

Output 2 - Schenley Park

Input Images
Description of GIF
COLMAP Reconstruction
Description of GIF

Important details for sparse reconstruction using COLMAP.

Procedure:
1. Set up the `images_directory` containing input images and `results_directory` for storing outputs. Define `target_resolution` to resize images if needed.
2. Create an output directory for resized images. For each image in `images_directory`, read and convert `.HEIC` files to RGB format.
3. Resize images to `target_resolution` if they exceed specified dimensions, then save each as `.jpg` in the output directory.
4. Set paths for `images_path`, `results_path`, and the COLMAP `database.db` file in `results_directory`.
5. Feature Extraction: Run `pycolmap.extract_features` on `database_file` and `images_path` to extract features from each image.
6. Feature Matching: Use `pycolmap.match_exhaustive` to perform exhaustive matching between extracted features.
7. Incremental Reconstruction: Run `pycolmap.incremental_mapping` for 3D reconstruction, saving the first reconstruction in `results_path`.
8. Print confirmation that the sparse reconstruction is complete and specify the output directory.

Key Steps and Outputs:
1. Image Resizing: Resize `.HEIC` images to `target_resolution` and convert to `.jpg`.
2. Feature Extraction and Matching: Extract image features and perform exhaustive feature matching.
3. Incremental Mapping: Perform incremental 3D reconstruction and save outputs in `results_path`.
4. Completion Message: Confirm sparse reconstruction completion and provide the results path for viewing.