Learning Based Image Synthesis
Assignment 1
Aviral Agrawal (avirala)
1. Overview
The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and,
using image processing techniques, automatically produce a color image with as few visual artifacts
as possible. We extract the three color channel images, place
them on top of each other, and align them so that they form a single RGB color image.
The naive approach to aligning the images is to use a simple x,y translation model over a window.
However, such an implementation becomes quartractically more expensive as the image size increases.
Thus, we need to use a more efficient approach to align the images. For this purpose, we will use
Gaussian Pyramid approach to downsample the images progressively and then align them using a big window.
Subsequently, we obtain the offsets and reduce to window size to align the images at a resolution
one-level higher. We repeat this process until we reach the original resolution.
For the purposes of similarity metric, we use Normalized Cross Correlation (NCC) to align the images.
However, upon close inspection, we notice that the NCC metric is not very effective for the image: Emir of Bukhara.
For that particular image, we compute NCC, not on the images themselves, but on the gradients of the images.
Furthermore, to demostrate the approach further, we colorize more images from the Prokudin-Gorskii dataset.
Lasty, we implement the same approach using Pytorch instead of numpy.
2. Single Scale NCC
The metric used for aligning the images is Normalized Cross Correlation (NCC). The NCC is computed as
$$\frac{\sum_{x,y} (f(x,y) - \bar{f})(g(i+x,j+y) - \bar{g})}{\sqrt{\sum_{x,y} (f(x,y) - \bar{f})^2 \sum_{x,y} (g(i+x,j+y) - \bar{g})^2}}$$
where 'f' is the template and 'g' is the image being aligned to.
As a pre-processing step throughout the assignment, we center crop the images to retain 75% of the original image.
This is done so as to compute the similarity metric only over the areas of interest and not over the
borders which have extensive artifacts.
For the cathedral image, we select a window of [-100, 100] since the image size is reasonably small.
Below in Figure 1. we show an example of why aligning the images is important.
Fig1. Cathedral colorized (unaligned)
Table 1. Aligned colorized Cathedral image and its offsets
| Image Name | Image |
|---|---|
| cathedral.jpg: g_offset : (5, 2), r_offset : (12, 3) |  |
3. Gaussian Pyramid NCC
In continuation of the quadratic increase in computation with increase in image size, we further discuss the
Gaussian Pyramid approach. The idea is to downsample the images progressively and then align them using a big window
of size [-100, 100]. Note that since the cathedral image is smaller, the window size if reduced to [-10, 10] so that
the window does not go out of bounds. Subsequently, we obtain the offsets and reduce to window size to align the images at a
resolution one-level higher. We repeat this process until we reach the original resolution.
This process helps us to fasten the process by several orders of magnitude. All the images generated below are after
downscaling the original images by a factor of 4.
Table 2. Aligned colorized images from provided dataset and their respective offsets
| Image Name | Image |
|---|---|
| cathedral.jpg: g_offset : (4, 2), r_offset : (12, 4) |  |
| harvesters.tif: g_offset : (60, 16), r_offset : (124, 14) |  |
| icon.tif: g_offset : (40, 16), r_offset : (90, 22) |  |
| emir.tif: g_offset : (48, 24), r_offset : (-772, -1640) |  |
| self_portrait.tif: g_offset : (78, 28), r_offset : (176, 36) |  |
| lady.tif: g_offset : (54, 8), r_offset : (116, 12) |  |
| train.tif: g_offset : (42, 4), r_offset : (88, 32) |  |
| three_generations.tif: g_offset : (52, 12), r_offset : (112, 12) |  |
| village.tif: g_offset : (64, 12), r_offset : (136, 22) |  |
| turkmen.tif: g_offset : (56, 20), r_offset : (116, 28) |  |
4. Better metric for Emir
From the results generated in section 3 for the image Emir of Bukhara, we notice that the NCC metric applied
on the image itself yields very poor results. At the heart of template matching, matching edges is a much better
alignment strategy than matching the pixel values since the latter is not robust to noise, etc.
In order to match edges, we compute the gradients of the image and the template using sobel filters and then
compute the NCC metric on the gradients.
From the results below, we can see that the alignment is much better than the previous approach.
Table 3. Better metric for Emir
| Image Name | Image |
|---|---|
| emir.tif: g_offset : (50, 24), r_offset : (106, 40) |  |
5. More images from Prokudin-Gorskii dataset
Table 4. Aligned colorized on more images and their respective offsets
| Image Name | Image |
|---|---|
| 1.tif: g_offset : (22, 16), r_offset : (146, 32) |  |
| 2.tif: g_offset : (-12, 2), r_offset : (22, 0) |  |
| 3.tif: g_offset : (30, 30), r_offset : (56, 46) |  |
| 4.tif: g_offset : (24, 20), r_offset : (60, 28) |  |