Q1: Camera matrix P from 2D-3D correspondences

(a) Stanford Bunny

1. Computed P matrix:

[[ 6.12468612e-01 -2.80769806e-01  1.09185025e-01  2.12092927e-01]
 [-8.90197009e-02 -6.43243106e-01  1.93261536e-01  1.73520830e-01]
 [ 5.51654830e-05 -1.35588807e-04 -7.00171505e-05  9.52266452e-05]]

1. Table of Surface points and Bounding Box line plots

Surface Points	Bounding Box

1. Method:

   • We are given 2D-3D correspondences. The first 2 elements of each correspondence is a 2D point, and the last 3 are the 3D point. We make sure that there at least 6 correspondences before proceeding.
   • Homogenize both the 2D and 3D point

   • We model the constraint as follows:

     

   • From the equation above, a single point correspondence gives us 2 constraints. We use 12 such constraints from 6 points to make matrix A of size (12 x 12)

   • The right null space of A (using SVD) gives us the best solution for the elements of the projection matrix.
   • The matrix multiplication (P @ X) projects a homogenized 3D point to the image plane. Converting to unit scale gives us the pixel location of that point. Using this, we are able to plot points and lines

(b) Cuboid

1. Computed P matrix

[[ 5.13165835e-03  7.24544382e-04 -1.12942559e-03  6.74930025e-01]
 [-1.93022997e-04  5.00213348e-04 -4.55849392e-03  7.37848188e-01]
 [ 8.61715030e-07 -9.99251293e-07 -6.38565796e-07  5.58604817e-04]]

1. Table of image, points annotated, and lines projected.

Input Image	Annotated 2D points	Example Result

1. Method:
   • We annotate the top left corner of the cube as the reference [0, 0, 0]. The following corners are annotated assuming a side length of 100 as shown in the annotation image.
   • Annotating 6 corners gives us 6 homogenized 3D-to-2D point correspondences and 12 constraints, similar to the equation used in Q1(a)
   • The right null space of A (using SVD) gives us the best solution for the elements of the projection matrix.
   • Choosing the points [50, 80] [35, 50] [65, 50] and [50, 20] gives us a pattern in the shape of a lightning bolt on the cube's face! The other lines are formed using the cube's corners.
   • The matrix multiplication (P @ X) projects a homogenized 3D point to the image plane. Converting to unit scale gives us the pixel location of that point. Using this, we are able to plot lines.

Q2: Camera calibration K from annotations

(a) Camera calibration from vanishing points

Input Image	Annotated Parallel Lines	Vanishing points and principal point

1. Intrinsics Matrix K:

[[ 1.15417802e+03  0.00000000e+00  5.75066005e+02]
 [-0.00000000e+00  1.15417802e+03  4.31939090e+02]
 [-0.00000000e+00  0.00000000e+00  1.00000000e+00]]

1. Vanishing points in pixel coordinates (x, y):

[[-1204.64633052  1425.62820743]
 [  559.88532351  -935.83692793]
 [ 1859.40405616  1391.62090484]]

1. Principal point in pixel coordinates (x, y):

[575.06600499 431.93909042]

1. Method:

   • We are given 3 parallel line pairs which are mutually orthogonal. We homogenize each 2D image point.
   • The cross product of each pair gives us the vanishing point of the line on the image. Convert the vanishing points to unit scale
   • From 3 vanishing points, each combination of 2 vanishing points gives us the following constraint, assuming zero skew and square pixels.

   • Using SVD to solve for the set of linear constraints, we get the image of the absolute conic (IAC)

     

   • The intrinsics K can be obtained by first doing a Cholesky decomposition of the IAC, and then taking the inverse of it.

   • The principal point is recovered from the last column of the intrinsics matrix K

(b) Camera calibration from metric planes

Input Image	Annotated Square 1	Annotated Square 2	Annotated Square 3

1. Intrinsics Matrix K

[[ 1.08447642e+03 -1.35121131e+01  5.20013594e+02]
 [-0.00000000e+00  1.07900526e+03  4.02544642e+02]
 [-0.00000000e+00  0.00000000e+00  1.00000000e+00]]

1. Method:

   • We are given annotations of the corner points of 3 squares. We homogenize each 2D image point.
   • For each square we compute the homography H that maps its corner points [[0, 0], [0, 1], [1, 1], [1, 0]] to the annotated points on the image. This is done using Direct Linear Transformation and SVD.
   • We plot each homography on the image using cv2 warp perspective.
   • We can form the following constraint using the homographies:

   • Thus, each homography gives us 2 constraints on the IAC. Since the IAC is symmetric (as shown in Q2(a)), we can find its elements by solving for the 6 constraints obtained from the 3 homographies we have. This is done using SVD.

     

   • The intrinsics K can be obtained by first doing a Cholesky decomposition of the IAC, and then taking the inverse of it.

   • Each square has two pairs of vanishing points, obtained from the cross product of its parallel line pairs.
   • The vanishing line for the plane that contains the square is obtained from the cross product of the vanishing points.
   • We obtain the plane normals by pre-multiplying the transpose of the intrinsics K.T with the vanishing line.
   • The angle between two planes is the angle between the normals. This is obtained from arccos(dot_product(normal_1, normal_2)), given that the normal vectors have unit magnitude.

(c) Camera calibration from rectangles with known sizes

Input Image	Annotated Squares	Annotated Plane 1	Annotated Plane 2	Annotated Plane 3

1. Intrinsics Matrix K

[[ 2.87309221e+03 -9.72158050e+00  2.03083377e+03]
 [-0.00000000e+00  2.89135314e+03  1.28359857e+03]
 [ 3.84542836e-19 -6.13569008e-19  1.00000000e+00]]

1. We use the exact same algorithm here as in Q2(b). The only difference is that we replace the corner points being mapped by the homography. Considering the ratio Rx:Ry between the x and y axes, we replace the corner points with [[0, 0], [0, Ry], [Rx, Ry], [Rx, 0]]. The rest of the algorithm is the same as Q2(b). In fact, Q2(b) is a special case of this where Rx:Ry = 1:1.

Q3: Single View Reconstruction

Note: I am using Open3D for just visualizing the 3D RGB points. This is because the matplotlib scatter plot window throws a "not responding" error when plotting a high number of points. Compared to this, Open3D has a much smoother visualizer for debugging and recording results.

Input Image	Annotations	Reconstruction View 1	Reconstruction View 2	Visualization

1. Method:

   • We are given the annotations of 3 mutually orthogonal planes.
   • We pick a parallel line pair from each of the 3 planes such that the the vanishing points for those lines satisfy the perpendicularity constraint.
   • Using the method in Q2(a), we can find the IAC from the 3 constraints given by the vanishing points of 3 mutually orthogonal planes (assuming zero skew and square pixels)
   • The intrinsics K can be obtained by first doing a Cholesky decomposition of the IAC, and then taking the inverse of it.
   • Each plane has two pairs of vanishing points, obtained from the cross product of its parallel line pairs.
   • The vanishing line for each plane is obtained from the cross product of the vanishing points.
   • We thus obtain the plane normals by pre-multiplying the transpose of the intrinsics K.T with the vanishing line.
   • The plane normal gives us 3 of the 4 values required to get the plane equation.
   • First, we backproject the central corner point that lies on all 3 planes, and set its scale to 1. Given that this 3D point lies on the plane, we get the equation of the plane.
   • For the other planes, using the annotation of an image point that lies on both a known plane and the new plane, we calculate the depth at which the ray intersects the known plane. Then we calculate the plane equation from this known 3D point.
   • To get a complete reconstruction, we find the pixels corresponding to each plane by interpolating a polygon between the plane corners.
   • We unproject each of these pixels into a ray, and find the scale at which the ray intersects its corresponding plane. This gives us the 3D point. We assign the RGB value of the pixel to this point. We visualize the plot using Open3D

   • Relevant equations given below:

     

(b) Similarly, we can use the same function for different images:

Input Image	Annotations	Visualization