Q1: Affine Rectification (30 points)¶
tiles5:
| Input Image | Annotated parallel lines on input image | Affine-Rectified Image | Test lines on Input Image | Test lines on Affine-Rectified Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.9868 | 0.9999 |
| 0.9987 | 0.9999 |
chess1:
| Input Image | Annotated parallel lines on input image | Affine-Rectified Image | Test lines on Input Image | Test lines on Affine-Rectified Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.9878 | 0.9997 |
| 0.9845 | 0.9999 |
facade:
| Input Image | Annotated parallel lines on input image | Affine-Rectified Image | Test lines on Input Image | Test lines on Affine-Rectified Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.7842 | 0.9999 |
| 0.9999 | 0.9999 |
chess (customized):
| Input Image | Annotated parallel lines on input image | Affine-Rectified Image | Test lines on Input Image | Test lines on Affine-Rectified Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.9841 | 0.9986 |
| 0.9846 | 0.9999 |
tiles (customized):
| Input Image | Annotated parallel lines on input image | Affine-Rectified Image | Test lines on Input Image | Test lines on Affine-Rectified Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.9943 | 0.9977 |
| 0.9998 | 0.9984 |
Implementation:¶
We first calculate the lines using pairs of points (Denote the lines as $l_1$ and $l_2$, $m_1$ and $m_2$). Then two vanishing points are computed by finding the intersection of lines $l_1$ & $l_2$, and $m_1$ & $m_2$. A line at infinity is determined by connecting the two vanishing points. Finally, the affine rectification matrix $H_{affine}$ is constructed using the line at infinity, following:
$$H_{\text{affine}} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ \frac{l_{\infty 0}}{l_{\infty 2}} & \frac{l_{\infty 1}}{l_{\infty 2}} & 1 \end{bmatrix} $$
Where $ l_{\infty 0} $, $ l_{\infty 1} $, and $ l_{\infty 2} $ are the components of the line at infinity, $l_{\infty}$.
For the provided images, I used the provided corresponding annotations. For the images I found, I annotated 4 pairs of parallel lines to calculate $H_{\text{affine}}$.
Q2: Metric Rectification (40 points)¶
tiles5:
| Input Image | Annotated perpendicular lines on input image | Annotated perpendicular lines on Affine-Rectified Image | Rectified Image | Test lines on Input Image | Test lines on Metric-Rectified Image |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.1677 | -0.0091 |
| 0.3498 | -0.0293 |
chess1:
| Input Image | Annotated perpendicular lines on input image | Annotated perpendicular lines on Affine-Rectified Image | Rectified Image | Test lines on Input Image | Test lines on Metric-Rectified Image |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.6685 | -0.0211 |
| -0.0448 | -0.0096 |
facade:
| Input Image | Annotated perpendicular lines on input image | Annotated perpendicular lines on Affine-Rectified Image | Rectified Image | Test lines on Input Image | Test lines on Metric-Rectified Image |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| -0.1919 | 0.0461 |
| -0.1065 | 0.0107 |
tiles (customized 1):
| Input Image | Annotated perpendicular lines on input image | Annotated perpendicular lines on Affine-Rectified Image | Rectified Image | Test lines on Input Image | Test lines on Metric-Rectified Image |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| -0.4855 | -0.0071 |
| -0.6774 | -0.0194 |
tiles (customized 2):
| Input Image | Annotated perpendicular lines on input image | Annotated perpendicular lines on Affine-Rectified Image | Rectified Image | Test lines on Input Image | Test lines on Metric-Rectified Image |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
| Before | After |
|---|---|
| 0.1687 | 0.0141 |
| 0.1194 | -0.0578 |
Implementation:¶
Given the affine rectification matrix, we can calculate the metric rectification matrix using only two pairs of perpendicular lines. Since the image is already affine-rectified, we know: $$ C^{*'}_{\infty} = \begin{bmatrix} a & b & 0 \\ b & c & 0 \\ 0 & 0 & 0 \\ \end{bmatrix} $$
We first transform the provided points using the affine rectification matrix $H_{affine}$. Homogeneous coordinates are normalized after the transformation. And then similar to Q1, we calculate the lines $l_1$ & $l_2$ and $m_1$ & $m_2$. A matrix $A$ is created using the line equations of the perpendicular lines. $$ A = \begin{bmatrix} l_1^0 \cdot l_2^0 & l_1^0 \cdot l_2^1 + l_1^1 \cdot l_2^0 & l_1^1 \cdot l_2^1 \\ m_1^0 \cdot m_2^0 & m_1^0 \cdot m_2^1 + m_1^1 \cdot m_2^0 & m_1^1 \cdot m_2^1 \end{bmatrix} $$
The matrix $A$ undergoes Singular Value Decomposition (SVD) to find the solution $s$. This solution is used to compute the circular points at infinity matrix $C^{*'}_{\infty}$ using the above equation. Another SVD is applied to $C^{*'}_{\infty}$, and the diagonal scaling matrix is used to compute the metric rectification matrix $H_{metric}$.
$$ H_{metric} = \text{diag} \left( \frac{1}{\sqrt{\sigma_0}}, \frac{1}{\sqrt{\sigma_1}}, 1 \right) \cdot U^T $$
The total transformation would be:
$$ H = H_{metric} H_{affine} $$
For the provided images, I used the provided corresponding annotations. For the images I found, I annotated 4 pairs of perpendicular lines to calculate $H_{metric}$.
Q3: Planar Homography from Point Correspondences (30 points)¶
| Normal Image | Perspective Image | Annotated corners in Perspective Image | Warped and Overlaid Image |
|---|---|---|---|
 |
 |
 |
 |
| Normal Image | Perspective Image | Annotated corners in Perspective Image | Warped and Overlaid Image |
|---|---|---|---|
 |
 |
 |
 |
Implementation:¶
We first determine the source points (the corners of the book cover) and the target points (the corners of the book on the desk). Taking these source points and target points as input, we can construct the matrix $A$ using the relationships between the points in the source image $(x, y)$ and their corresponding points in the destination image $(x^{'}, y^{'})$. These relationships are expressed as:
$$ \begin{aligned} x' &= \frac{h_{11}x + h_{12}y + h_{13}}{h_{31}x + h_{32}y + h_{33}} \\ y' &= \frac{h_{21}x + h_{22}y + h_{23}}{h_{31}x + h_{32}y + h_{33}} \end{aligned} $$
By rearranging these equations, the following two linear equations are derived for each pair of points $(x, y)$ and $(x^{'}, y^{'})$:
$$ -x h_{31} - y h_{32} - h_{33} + x' (h_{11}x + h_{12}y + h_{13}) = 0 $$
$$ -x h_{21} - y h_{22} - h_{23} + y' (h_{11}x + h_{12}y + h_{13}) = 0 $$
Each point contributes two equations, resulting in a system of linear equations that form the matrix $A$. The structure of the matrix for a single point pair looks like this:
$$ \begin{aligned} [-x, -y, -1, 0, 0, 0, x \cdot x', y \cdot x', x'] \\ [0, 0, 0, -x, -y, -1, x \cdot y', y \cdot y', y'] \end{aligned} $$
We then use SVD to solve for the homography matrix $H$. The homography matrix H corresponds to the last column of the matrix $V$ (the right singular vectors). The resulting 9-element vector from V is reshaped into a 3x3 matrix, which is the homography matrix H:
$$ H = \begin{bmatrix} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{bmatrix} $$
This matrix can then be used to warp the source image to align with the destination image.
For the provided image pairs, I used the provided annotations. For the image pairs that I found, I annotated the four corners of the book on the desk.
Q4: Bonus: Metric Rectification from Perpendicular Lines (10 points)¶
| Input Image | Annotated perpendicular lines | Rectified Image |
|---|---|---|
 |
 |
 |
| Before | After |
|---|---|
| 0.1498 | -0.0049 |
| 0.1809 | -0.0192 |
| -0.1643 | -0.0401 |
| Input Image | Annotated perpendicular lines | Rectified Image |
|---|---|---|
 |
 |
 |
| Before | After |
|---|---|
| 0.2273 | 0.0116 |
| -0.4239 | 0.0018 |
| 0.5440 | -0.0054 |
Implementation:¶
Similar to Q2, we can denote $ C^{*'}_{\infty} = \begin{bmatrix} a & b & d \\ b & c & e \\ d & e & f \\ \end{bmatrix} $ since the image is not already affine-rectified.
For each pair of perpendicular lines, we can derive an equation that $$ l_0 m_0 a + (l_0 m_1 + l_1 m_0) b + l_1 m_1 c + (l_0 m_2 + l_2 m_0) d + (l_1 m_2 + l_2 m_1) e + l_2 m_2 f = 0 $$
So we can use 5 pairs of perpendicular lines to construct a matrix $A$. The matrix $A$ then undergoes SVD, which decomposes it into three matrices ($U$, $S$, $Vt$). Still, the last row of the matrix $Vt$ ($s$) contains the parameters used to build $C^{*'}_{\infty}$. Another SVD is applied to $C^{*'}_{\infty}$, and the diagonal scaling matrix is used to compute the metric rectification matrix $H_{metric}$.
$$ H_{metric} = \text{diag} \left( \frac{1}{\sqrt{\sigma_0}}, \frac{1}{\sqrt{\sigma_1}}, 1 \right) \cdot U^T $$
Q5: Bonus: More Planar Homography from Point Correspondences (10 points)¶
| Image 1 | Image 2 | Image 3 | Perspective Image | Warped and Overlaid Image |
|---|---|---|---|---|
 |
 |
 |
 |
 |
Implementation:¶
We can use the same algorithm in Q3 to achieve this effect. In Q3, we only have 4 points in the annotation. But in Q5, we could have more points (in my example, I have 12 points in total, corresponding to 3 images). So I annotated them in order, and applied similar code to get the final result.