# Assignment 1: Projective Geometry and Homography
## Question 1: Affine Rectification
To do affine rectification, we need first use annotation to find two pairs of parallel lines in the image. Then, we can do following steps:
- Represent the lines using two points from annotations:
- $\bf{l} = \bf{x} \times \bf{y}$
- Find the vanishing points of each lines:
- $\bf{v} = \bf{l_1} \times \bf{l_2}$
- Find the infinity line using two infinite points:
- $\bf{l_{\inf}} = \bf{v_1} \times \bf{v_2}$
- The H can be represented as: $\bf{l_{\inf}} = H^{-T}l_{\inf}^{'}$
$$ H = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
l_1 & l_2 & l_3
\end{bmatrix}$$
### Image 1
| Input Image | Annotated parallel lines| Affine-Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.896445 | 0.99999 |
| 0.957452 | 0.99997 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Image 2
| Input Image | Annotated parallel lines| Affine-Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.784232 | 0.999978 |
| 0.999997 | 0.999986 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Image 3
| Input Image | Annotated parallel lines| Affine-Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.986833 | 0.999925 |
| 0.998719 | 0.999938 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
## Self-Captured Image 1
| Input Image | Annotated parallel lines| Affine-Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.999230 | 0.999741 |
| 0.956036 | 0.999165 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
## Self-Captured Image 2
| Input Image | Annotated parallel lines| Affine-Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.999317 | 0.999999 |
| 0.999152 | 0.999997 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
## Question 2: Metric Rectification
To do metric rectification in two steps, we will need the result from affine rectification, and two more pairs of orthogonal lines. Then, we can do following steps:
- Represent the lines using two points from annotations as before.
- Construct the conic matrix:
$$
C = \begin{bmatrix}
a & \frac{b}{2} & 0 \\
\frac{b}{2} & c & 0 \\
0 & 0 & 0
\end{bmatrix}
$$
- At this time, $d, e, f = 0$ since the image is already affine-rectified.
- Construct the function $A\bf{c} = 0$:
- each row of $A$ is $a_i = \begin{bmatrix}x_1\times x_2 & \frac{1}{2}\times(x_1\times y_2 + x_2\times y_1) & y_1 \times y_2\end{bmatrix}^T$
- Then use SVD to solve the $C_{\inf}^{'}$ matrix.
- To solve $H$, we have: $C_{\inf} = H C_{\inf}^{'} H^{T}$, then
$$
H = \begin{bmatrix}
\sqrt{\sigma_{1}^{-1}} & 0 & 0 \\
0 & \sqrt{\sigma_{2}^{-1}} & 0 \\
0 & 0 & 1
\end{bmatrix} U^{T}
$$
where $\sigma{1}$ and $\sigma{2}$ are the singular values of $C_{\inf}^{'}$.
### Image 1
| Input Image | Annotation on input image | Annotation on Affine-Rectified Image | Rectified Image |
| ----------- | ----------- | ----------- | --------- |
| | 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| -0.252671 | 0.001817 |
| 0.0881734 | 0.008199 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Image 2
| Input Image | Annotation on input image | Annotation on Affine-Rectified Image | Rectified Image |
| ----------- | ----------- | ----------- | --------- |
| | 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| -0.191871 | 0.046072 |
| -0.106548 | 0.010747 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Image 3
| Input Image | Annotation on input image | Annotation on Affine-Rectified Image | Rectified Image |
| ----------- | ----------- | ----------- | --------- |
| | 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.167718 | -0.008092 |
| 0.034983 | -0.029307 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Self-Captured Image 1
| Input Image | Annotation on input image | Annotation on Affine-Rectified Image | Rectified Image |
| ----------- | ----------- | ----------- | --------- |
| | 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.166180 | -0.055056 |
| -0.137629 | 0.090029 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
### Self-Captured Image 2
| Input Image | Annotation on input image | Annotation on Affine-Rectified Image | Rectified Image |
| ----------- | ----------- | ----------- | --------- |
| | 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| 0.058722 | -0.013858 |
| -0.029024 | 0.016103 |
#### Test Annotation
| Test lines on Input Image | Test lines on Affine-Rectified Image |
| ----------- | ----------- |
| 
| 
|
## Question 3: Planar Homography from Points Correspondences
To do planar homography, we need to find four corner points of the normal image and the corresponding points in the perspective image. Then, we can do following steps:
- Using all the corresponding points, we can calculate $H$ based on the equation: $H\bf{x}\times\bf{x}^{'} = 0$. This could convert to:
$$
[\bf{x}^{T}]_{\times} \begin{bmatrix}
\bf{x}^{T} & \bf{0} & \bf{0} \\
\bf{0} & \bf{x}^{T} & \bf{0} \\
\bf{0} & \bf{0} & \bf{x}^{T}
\end{bmatrix} \bf{h} = 0
$$
It can be observed that the third row is not independent, which could be represented using the first two rows. Therefore, we have:
$$
\begin{bmatrix}
\bf{0} & -w^{'}\bf{x}^{T} & y^{'}\bf{x}^{T}\\
w^{'}\bf{x}^{T} & \bf{0} & -x^{'}\bf{x}^{T}
\end{bmatrix} \bf{h} = 0
$$
- The we can construct the matrix $A$:
- $A$ is a $(2\cdot4)\times 9$ matrix, where each corresponding point can provide two constraints, and they are represented as above.
- Use SVD to solve the $A$ matrix and reshape to $H$.
### Image
| Input Image | Annotated perpendicular lines | Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
### Self-Captured Image
| Input Image | Annotated perpendicular lines | Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
## Question 4: Bonus: Metric Rectification from Perpendicular Lines (10 points)
To do metric rectification from perpendicular lines in one step, we need to anntate at least 5 pairs of perpendicular lines in the image. Then, we can do following steps:
- Represent the lines using two points from annotations as before.
- Construct the conic matrix:
- At this time, the Conic $C$ has 6 Dof, which can be represented as:
$$
C = \begin{bmatrix}
a & \frac{b}{2} & \frac{d}{2} \\
\frac{b}{2} & c & \frac{e}{2} \\
\frac{d}{2} & \frac{e}{2} & f
\end{bmatrix}
$$
- Then, similar as above, we can construct the function $A\bf{c} = 0$: (I use the order: abdcef for better calculate)
- each row of $A$ is $a_i = \begin{bmatrix}x_1x_2 & \frac{1}{2}\times(x_1y_2 + x_2y_1) & \frac{1}{2}\times(x_1z_2 + x_2z_1) & y_1 y_2 & \frac{1}{2}\times(y_1z_2 + y_2z_1) & z_1 z_2\end{bmatrix}^T$
- Also, use SVD to solve the $C_{\inf}^{'}$ matrix. Then, we can solve $H$ as above.
### Image
| Input Image | Annotated perpendicular lines | Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| -0.054563 | 0.008889 |
| -0.054504 | 0.024648 |
| -0.167346 | -0.046538 |
| 0.006097 | -0.071508 |
| 0.142309 | -0.009877 |
### Self-Captured Image
| Input Image | Annotated perpendicular lines | Rectified Image |
| ----------- | ----------- | ----------- |
| 
| 
| 
|
#### Evaluate Angle
| Before | After |
| -----------|-----------|
| -0.084163 | 0.023170 |
| 0.057618 | 0.003299 |
| -0.222243 | -0.001927 |
| 0.007773 | 0.008215 |
| 0.120606 | 0.006619 |
## Q5: Bonus: More Planar Homography from Point Correspondences (10 points)
To do this, we need to do serveral times of planar homoography just like Q3. I used 6 images to generate this new photo of the **Times Square**. Hope you like it!
### Input Image
### Output Image
## How to run
- main.py:
- python main.py [-q] [0, 1, 2, 3, 4, 5] [-r] [root_path]
- When you choose q=0, it will run all the questions, otherwise, it will run the question you choose.
- For the root path, it should be structured like this:
- root_path
- q1
- ...
- q3
- ...
- q4
- ...
- annotation
- ...
- utils.py:
- python utils.py [-q] [q1, q2, q3, q4, q5] [-r] [root_path]
- The root path structure should be the same as above.