Overview
The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, the three color channel images are extracted from the digitzed glass tile, aligned and stacked on top of each other, so that they form a single RGB color image.
Approach
The easiest way to align the channels is to exhaustively search over a widow of possible translations, scoring each move based on some image matching metric. For the purpose of this assignment, I explored two simple metrics: SSD (sum of squared differences) and NCC (normalized cross correlation).
\[ SSD[m,n] = \sum_{x,y} (I_1(x,y) - I_2(x+m,y+n))^2 \]
\[ NCC[m,n] = \frac{\sum_{x,y} (I_1(x,y) - \mu_1)(I_2(x+m,y+n) - \mu_2)}{\sqrt{\sum_{x,y} (I_1(x,y) - \mu_1)^2 \sum_{x,y} (I_2(x+m,y+n) - \mu_2)^2}} \]
A single scale implementation with the above metrics works perfectly for small resolution images, but the time complexity of the exhaustive search over high resolution images is too high with compute time exceeding 10 mins at times. For dealing with high-res images, a multi-scale pyramid approach is used.
and a coarse alignment is first performed on the smallest image.
These estimated translations are then scaled up appropriately for
the next level and the grid search is repeated around those values.
Thus the translation is iteratively refined at each level
until we reach the original resolution. This signifactly reduces the
runtime as we can run smaller grid search at larger resolutions.
Results
I observed exact same offsets from both SSD and NCC metrics. Since SSD was faster to run, all results seen below are using SSD. Also, to reduce the border artifacts from the original glass tile images, I used a fixed left and right crop of 5% on all images.
1. Single scale alignment
(3, 12)(2, 5)2. Pyramidal alignment results
(23, 89)(17,40)
(10, 11)(14, 53)
(13, 124)(16, 60)
(-403, 95)(24, 49)
(22, 137)(12, 65)
(11, 113)(8, 47)
(36, 176)(28, 78)
(26, 115)(18, 55)
(31, 87)(5, 42)Similar results can be observed on some of my handpicked images from the online LoC collection
(-4, 134)(1, 62)
(31, 81)(-57, 0)
(-26, 122)(-14, 34)
(-26, 128)(-12, 60)
(16, 11)(10, -17)Emir and Riverside are the two images for which SSD alignment fails, primarily because they high illumination variance between the channels. Emir can be fixed by using weighted SSD as a metric and Riverside can be fixed using SIFT features for alignment instead of raw pixel values. Both of these approaches are discussed in the Bells and Whistles section.
Bells and Whistles
1. Weighted SSD Loss
Since np.roll is used, image borders are the major source of error when calculating SSD. Also, when aligning images like Emir where the channels have
large differences in brightness, the SSD loss is dominated by a small percentage of mismatched pixels. If we reduce the contribution
of these pixels in the loss term we can get a better alignment. The result of such a metric is shown below.
2. Better features and better alignment
Aligning based on RGB similarity is not a robust strategy. A better approach is to calculate SIFT features and align the corresponding keypoints based on their descriptors.
The correspondence can sometimes be noisy and Lowe's ratio test is used to filter out the bad matches. A homography can then
be estimated from the good matches and thus one channel can be warped onto the other channel for alignment. I used OpenCV's
SIFT implementation (its highly optimized and runs faster than pyramid alignment even at native resolutions) as well
as cv2.findHomography and cv2.warpPerspective for final alignment.
This method outperforms the RGB based alignment and improvements can be clearly seen on Emir and Riverside images.
Feature based alignment works perfectly on other images aswell and gives almost similar translations.
(18, 79)(14,31)
(7, 96)(8, 34)
(15, 130)(4, 54)
(-17, 80)(5, 52)
(33, 160)(25, 64)
(32, 108)(14, 43)3. Better transformations
As described in the previous section, having feature correspondences allows us to directly estimate the homography between the two images. Thus we get a full projective transform (8 DOFs) instead of just a 2 DOF translations. It also takes into account any affine transformation that might be present in the images with almost no overhead in computation. The benefit is small for our dataset as simple translation is enough, but it can be useful for other images.
4. Automatic Contrasting
Automatic contrasting may help in improving the perceived image quality, especially in images with low contrast.
Two common apporaches are contrast stretching and histogram equalization.
Contrast Stretching : The image is rescaled to include all intensities that fall within the 2nd and 98th percentiles.
Histogram Equalization : This operation spreads out the most frequent intensity values in an image.
Histogram equalization seems to give a great result in case of Turkmen.tiff but produces unrealistic image in case of Emir.tiff
Challenges
Border clipping: The entire process of alignment using SSD rather than feature based matching was
very sensitive to border artifcats and I had to try out a lot of cropping strategies before settling on uniform 5% crop.
Number of levels in pyramid: Apparently if the resized image is too small [256 px] the coarse alignment suffers
and it cascades down to the final alignment.
Emir: Nothing seemed to work on him until I accidentally tried weighted SSD.