This is my webpage submission for 16822 assignment 1.

Q1: Affine Rectification

I performed affine rectification on 3 provided images and 2 images that I captured and annotated myself. An affine transformation is one that preserves parallel lines, which is to say that it preserves the line at infinity before and after the transformation. With annotations of lines that should be parallel, I mapped two pairs of would-be parallel lines into the 2D projective space and found their intersections. In Euclidean space, these lines should not intersect, but in the 2D projective space, they intersect at the line at infinity. Using the two intersection points from both pairs of lines, I was able to dervie the line at infinity of the form $l = (l_1, l_2, l_3)^T$. The transformation matrix to rectify the line at infinity back to $l_\infty = (0, 0, 1)^T$ is then

$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ l_1 & l_2 & l_3 \end{pmatrix}$

To evaluate the performance of my affine rectification, I calculated cosines between pairs of parallel lines that were not used in determining the rectification transformation. Parallel lines should have a cosine of 1, and the following table shows improvements towards this target when moving from before to after rectification.

Affine rectification using the provided tiles5 image:

Test line cosines for tiles5:

Affine rectification using the provided facade image:

Test line cosines for facade:

Affine rectification using the provided tiles3 image:

Test line cosines for tiles3:

Affine rectification using the captured mario image:

Test line cosines for mario:

Affine rectification using the captured squirrel image:

Test line cosines for squirrel:

Q2: Metric Rectification

I performed metric rectification on 3 provided images and 2 images that I captured and annotated myself. This work built upon the affine rectification performed in Q1, and took advantage of only requiring two pairs of annotated lines that should be perpendicular. Whereas in Q1 I found a transformation that preserve the line at infinity, for metric rectification I was interested in preserving the conic dual to the circular points, $C_\infty^* \prime = HC_\infty^*H^T$. Similar to how I compute the line at infinity and then mapped it back to its proper value in Q1, I will do the same with the dual conic here, where $C_\infty^* = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}$.

This problem was solved using two rounds of SVD. In the first round, I was solving for the symmetrical 2x2 matrix S, which represents the nonzero upper 2x2 block of $C_\infty^* \prime$. Using perpendicular lines $l$ and $m$, this problem takes the form $\begin{pmatrix} l_1 & l_2 & l_3 \end{pmatrix} \begin{pmatrix} S_{11} & S_{12} & 0 \\ S_{12} & S_{22} & 0 \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} m_1 \\ m_2 \\ m_3 \end{pmatrix}$, and solving further gives us the 1x3 matrix $\begin{pmatrix} m_1l_1 & m_1l_2 + m_2l_1 & m_2l_2 \end{pmatrix}$ for each pair of lines. Using two sets of perpendicular lines therefore gives us a 2x3 matrix to solve with SVD.

Finally, I ran a second SVD on the full, 3x3 form of $C_\infty^* \prime$ to obtain its principal values. From here, I determined the metric rectification transformation to be H = $\begin{pmatrix} \sqrt{\sigma^{-1}} & 0 & 0 \\ 0 & \sqrt{\sigma^{-1}} & 0 \\ 0 & 0 & 1 \end{pmatrix}$

To evaluate the performance of my metric rectification, I calculated cosines between pairs of perpendicular lines that were not used in determining the rectification transformation. Perpendicular lines should have a cosine of 0, and the following table shows improvements towards this target when moving from before to after rectification.

Metric rectification using the provided tiles5 image:

Test line cosines for tiles5:

Metric rectification using the provided chess1 image:

Test line cosines for chess1:

Metric rectification using the provided checker1 image:

Test line cosines for checker1:

Metric rectification using the captured mario image:

Test line cosines for mario:

Metric rectification using the provided mydesk image:

Test line cosines for mydesk:

Q3: Planar Homography from Point Correspondences

I computed homographies between two images using point correspondences, first with the provided images and then with a pair of images that I captured and annotated. I did not annotate the normal images, but rather used their corner coordinates $(0,0), (width, 0), (width, height), (0, height)$ to obtain the second pair of coordinates that made-up my correspondences.

Leveraging the DLT algorithm that we derived in class and saw on problem set 1, I set-up my constraints. Each set of points gave me two constraints and took the form of a 2x9 matrix. With four point correspondences, this gave me an 8x9 matrix to solve via SVD. Specifically, the equation that I solved was:

$\begin{pmatrix} w'\textbf{x}^T & \textbf{0}^T & -x'\textbf{x}^T \\ \textbf{0}^T & -w'\textbf{x}^T & y'\textbf{x}^T \end{pmatrix} \begin{pmatrix} \textbf{h}^1 \\ \textbf{h}^2 \\ \textbf{h}^3 \end{pmatrix} = \textbf{0}$, where the h terms represent the columns of the 9-element homography matrix (stretched into a 9x1 matrix), the prime terms ($x', y', w'$) are from the normal image and the vector term $\textbf{x}^T$ is from the perspective image.

Finally, I used a mask and OpenCV's warpPerspective function to overlay the warped normal image onto the correct location in the perspective image.

Planar homography using the provided desk-perspective and desk-normal images:

Planar homography using the captured mydesk-perspective and mydesk-normal images:

Q4: Metric Rectification from Perpendicular Lines

Adapting the metric rectification performed in Q2 to work without a first affinely rectified image involved the use of 5 pairs of perpendicular lines instead of 2.

A pair of lines produced the following 1x6 constraint, and stacking five of these together gave me a 5x6 matrix to solve via two rounds of SVD, similar to Q2. In the calculation of my dual conic, however, I used all computed values a, b, c, d, e, and f instead of only calculating the upper-left 2x2 block.

For two lines $l$ and $m$, the constraints that I used were of the form $\begin{pmatrix} l_1m_1, \frac{(l_1m_2 + l_2m_1)}{2}, l_2m_2, \frac{l_1m_3 + l_3m_1}{2}, \frac{l_2m_3 + l_3m_2}{2}, l_3m_3 \end{pmatrix}$

Q5: More Planar Homography from Point Correspondences

To expand the planar homography exercise from Q3, I took a photo with perspective of my living room and computed 3 separate homographies to overlay video game cover art over flat surfaces. The theory underlying this implementation was identical to the code used in Q3. For this question, I created the homography function in my own Python file to compute homographies without excessive copy-and-paste.