16-822: Geometry Based Methods in Vision¶
Assignment 1: Projective Geometry and Homography¶
Q1: Affine Rectification¶
Algorithm:¶
For affine rectification, we require two pairs of parallel lines. We use the first 8 points in the provided annotations and use their cross product to get 2 pairs of parallel lines $\mathbf{l}_1, \mathbf{l}_2$ and $\mathbf{l}_3, \mathbf{l}_4$.
We calculate the intersection of these two lines using their cross product giving us two points at infinity $\mathbf{p}_1, \mathbf{p}_2$. We find the line at infinity by using $l_{\infty} = \mathbf{p}_1 \times \mathbf{p}_2$.
For affine rectification, the find the transformation which restores the line at infinity $l_{\infty}$. This is given by the following where $l_{\infty} = \begin{bmatrix}a & b & c\end{bmatrix}^T$
$$
\mathbf{H}_\text{affine} =
\begin{pmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
a & b & 1
\end{pmatrix}
$$
This transformation is applied to the image to get the final rectified image.
Results:¶
1. Chess¶
| Input Image |
Annotated Lines |
Affine Rectified Image |
 |
 |
 |
| Before |
After |
| -0.9878 |
0.9845 |
| -0.9997 |
0.9999 |
| Test Lines on Input |
Test Lines on Affine Rectified |
 |
 |
2. Facade¶
| Input Image |
Annotated Lines |
Affine Rectified Image |
 |
 |
 |
| Before |
After |
| 0.7842 |
0.9999 |
| 0.9999 |
0.9999 |
| Test Lines on Input |
Test Lines on Affine Rectified |
 |
 |
3. Tiles¶
| Input Image |
Annotated Lines |
Affine Rectified Image |
 |
 |
 |
| Before |
After |
| 0.9868 |
0.9999 |
| 0.9987 |
0.9999 |
| Test Lines on Input |
Test Lines on Affine Rectified |
 |
 |
4. Custom Image 1¶
| Input Image |
Annotated Lines |
Affine Rectified Image |
 |
 |
 |
| Before |
After |
| 0.4791 |
0.9999 |
| 0.9999 |
0.9999 |
| Test Lines on Input |
Test Lines on Affine Rectified |
 |
 |
4. Custom Image 2¶
| Input Image |
Annotated Lines |
Affine Rectified Image |
 |
 |
 |
| Before |
After |
| 0.9977 |
0.9999 |
| 0.9924 |
0.9999 |
| Test Lines on Input |
Test Lines on Affine Rectified |
 |
 |
Q2. Metric Rectification¶
Algorithm:¶
For metric rectification, we require two pairs of perpendicular lines if affine rectification is already done. We use the first 8 points in the provided annotations to get the perpendicular lines. Let the two lines be $l_1', m_1'$ and $l_2', m_2'$.
For perpendicularity, we must have $l_i' C^{*'}_\infty m_i = 0$. If affine rectification is already done, $C^{*'}_\infty$ has the following form:
$$
C^{*'}_\infty = \begin{bmatrix} a & b/2 & 0 \\ b/2 & c & 0 \\ 0 & 0 & 0\end{bmatrix}
$$
Using the above relation, we can rewrite the constraint as $Ah = 0$ where $h$ is a flattened version of $C^{*'}_\infty$. This can be solved using SVD where the singular vector corresponding to the smallest singular value gives us $h$. To get the metric rectification transform, we now need to find $H$ such that:
$$
C^{*'}_\infty = H \cdot C^{*}_\infty \cdot H^T
$$
Using the fact that $C^{*'}_\infty$ is a symmetric matrix of rank 2, we can get its singular value decomposition as $U, S, U^T = svd(C^{*'}_\infty)$. From this, we can obtain $H$ as follows:
$$
H = \begin{bmatrix} 1/\sqrt{s1} & 0 & 0 \\ 0 & 1/\sqrt{s2} & 0 \\ 0 & 0 & 1 \end{bmatrix} \cdot U^T
$$
The obtained H can then be used to warp the images
Results¶
1. Chess¶
| Input Image |
Perpendicular Lines on Input Image |
Perpendicular Lines on Affine Rectified Image |
Rectified Image |
|
 |
 |
 |
| Before |
After |
| -0.6685 |
0.0211 |
| 0.0447 |
0.0096 |
| Test Lines on Input |
Test Lines on Metric Rectified |
 |
 |
2. Facade¶
| Input Image |
Perpendicular Lines on Input Image |
Perpendicular Lines on Affine Rectified Image |
Rectified Image |
|
 |
 |
 |
| Before |
After |
| 0.1918 |
-0.0460 |
| -0.1065 |
0.0107 |
| Test Lines on Input |
Test Lines on Metric Rectified |
 |
 |
3. Tiles¶
| Input Image |
Perpendicular Lines on Input Image |
Perpendicular Lines on Affine Rectified Image |
Rectified Image |
|
 |
 |
 |
| Before |
After |
| -0.1677 |
0.0080 |
| -0.0349 |
0.0293 |
| Test Lines on Input |
Test Lines on Metric Rectified |
 |
 |
4. Custom Image 1¶
| Input Image |
Perpendicular Lines on Input Image |
Perpendicular Lines on Affine Rectified Image |
Rectified Image |
|
 |
 |
 |
| Before |
After |
| 0.7838 |
0.0183 |
| 0.6092 |
-0.0066 |
| Test Lines on Input |
Test Lines on Metric Rectified |
 |
 |
4. Custom Image 2¶
| Input Image |
Perpendicular Lines on Input Image |
Perpendicular Lines on Affine Rectified Image |
Rectified Image |
|
 |
 |
 |
| Before |
After |
| 0.2921 |
-0.02492 |
| 0.1483 |
-0.0327 |
| Test Lines on Input |
Test Lines on Metric Rectified |
 |
 |
Q3: Estimating Homography¶
Algorithm:¶
To estimate the homography matrix $\mathbf{H}$, we require four point correspondences between the two images. We use the 4 provided corner annotations along with the corner points of the template image as the four correspondences.
To avoid the influence of the scale of pixel coordinates, we can normalize the point locations by ensuring that the centroid of the points is the origin and the max distance from origin is $\sqrt{2}$. Let the normalization matrices for the two sets of points be $T_1$ and $T_2$.
Given four point correspondences $(\mathbf{x}_i, \mathbf{x}'_i)$ for $i = 1, 2, 3, 4$, the homography matrix $\mathbf{H}$ is computed by solving:
$$
\mathbf{x}'_i = \mathbf{H} \mathbf{x}_i \quad \text{for} \quad i = 1, 2, 3, 4
$$
This gives us a set of linear equations $ \mathbf{x}'_i \times (\mathbf{H} \mathbf{x}_i) = \mathbf{0} $. We can stack these equations for all points and solve for $\mathbf{H}$ using singular value decomposition (SVD). The singular vector corresponding to the smallest eigenvalue gives us $H$.
We can unnormalize the homography matrix using $H_unnormalized = $T_2^T \cdot H \cdot T_1$.
| Normal Image |
Perspective Image |
Annotated Corners |
Warped and Overlaid |
 |
 |
 |
 |
 |
 |
 |
 |
Q4: Metric Rectification using Perpendicular Lines¶
Algorithm:¶
For this, we use the same algorithm as Q2. The only difference is that $C^{*'}_\infty$ has the following form:
$$
C^{*'}_\infty = \begin{bmatrix} a & b/2 & d/2 \\ b/2 & c & e/2 \\ d/2 & e/2 & f\end{bmatrix}
$$
| Input Image |
Annotated Perpendicular Lines |
Rectified Image |
|
 |
 |
| Before |
After |
| -0.1576193869 |
0.0081132557 |
| -0.0343966486 |
0.0203824362 |
| -0.0638538154 |
0.0295612319 |
| Input Image |
Annotated Perpendicular Lines |
Rectified Image |
|
 |
 |
| Before |
After |
| 0.221031522 |
0.1010012917 |
| -0.2808394841 |
0.2122132805 |
| -0.4112591831 |
0.0360628597 |
Q5: Stats overlay¶
Sports broadcasts often have overlays of stats on the screen. The following is an example of a such an overlay - a shot map in a football match. This shows the from where shots were taken and where they ended up (the streaks were added on top manually for completeness). Additionally, one of the boards behind the goal post has been replaced by another. </br>
</br>
</br>
</br>
</br>