inesr16726

Inés Rodríguez Hsu

Project Overview

In the early 1900s Prokudin-Gorskii took a series of black and white photographs which he envisioned to become color photographs by shooting three exposures of every photograph onto a glass plate using a red, a green and a blue filter. In this assignment I write a program to produce a color image for the digitized Prokudin-Gorskii glass plate images using image processing techniques. This involved aligning the three color channel images on top of each other to produce an RGB color image.

Implementation Approach

I implemented two different methods to achieve color images from Prokudin-Gorskii glass plate images.

Method 1:

I first obtained the three individual color channel images by divided the glass plate image into parts of three equal height, where the top image is the blue channel image, the middle is the red, and the bottom is the green. I cropped the three images by 50 pixels on each side of the image to cut out the borders, as those can affect the alignment. Then I try align the red and the green channel images to the blue channel image. I do this by searching over a window of possible horizontal and vertical displacements displacements (from -15 to 15) and scoring the alignment match of the images based on the Sum of Squared Differences (SSD) distance metric. This metric gives a measure of similarity or distance between two images according to the following formula, where I and P are two images, and the sum is being computed over the pixel values of the images. The lowest score indicates highest similarity, thus is the best score of displacement.

Photograph of Sum of Squared Differences formula

I apply a translation of the best scoring displacement for green and red color channel images and overlay them onto the blue image to produce the RGB color image. To make my code run efficiently I use the numpy vectorize several operations. I first obtain an array of all possible [x,y] displacement combinations. I obtain an array of all possible displaced images through a vectorized function which rolls the original image by all displacement combinations (specified by the combinations array) concurrently. I then compute the sum of square differences distance score for each displaced image compared to the original image also using a vectorized function to calculate the SSD for all displacements concurrently. This method works for smaller images, yet when applied to large images, for which the displacement window would have to be larger, it can be very computationally expensive to search through all the displacement windows.

Method 2:

For larger images, I modified the method detailed above to a more efficient method. As these images are larger, I cropped the three color channel images by 200 instead of 50, and I used a larger window of displacement combinations, starting from [-30,30]. I first made an image pyramid for all three color channel images. I made image pyramids of 6 levels (the first being the original image), where the image of each level is half the size of the previous level. To obtain each subsequent level I first apply a gaussian blur filter to the previous image through a convolution of a 3x3 Gaussian kernel and the image. This is to reduce aliasing effects in downsampling. Then I sample the pixels of every other row and column of the image to scale the image by a factor of two. I then start by applying the previous method to smallest or most coarse image, obtaining all possible displacement images and finding the best sum of squared differences distance score. I then apply that displacement to each image in the pyramid, scaling the displacement by 2 for each subsequent image. I then go down one level in the pyramid and do the same as for the first, but with a window of displacements that is smaller, and to adjust the displacement obtained in the previous level. I do the same for all levels, decreasing the window for each level, to obtain best overall displaced image for red and green color channel images. I again overlay all three color channel images to obtain the RGB color image.

An example of an image at each level is given below (with green and red images layered over blue image):

Results: Example Images and Offsets

The following are the results obtained for the Prokudin-Gorskii glass plate images given in the assignment. The first image (Cathedral) was obtained using just the first method, while the rest are obtained using the second, as they are larger images. The Prokudin-Gorskii glass plate images are on the left, and the RGB color produced is on the right. The displacement is given as [vertical displacement, horizontal displacement] in pixels. Overall results obtained were satisfactory. Images where the displacement was inaccurate were the 'Three generations' image and the 'Emir' image, as discussed below.

Cathedral:

Green displacement: [5,2]

Red displacement: [12,3]

Icon:

Green displacement: [40,17] Red displacement: [89,23]

Harvesters:

Green displacement: [59,16]

Red displacement: [124,13]

Lady:

Green displacement: [47,8]

Red displacement: [113,11]

Self-portrait:

Green displacement: [79,30]

Red displacement: [176,38]

Three generations:

Green displacement: [-114,0]

Red displacement: [-54,0]

The results obtained for this image were unsatisfactory. The displacement of the images worsened progressively as shown below. This image compared to others seemed to have larger flat spaces that looked the same, which could have contributed to less acurate results as maybe there was little change in the the SSD score for these flat regions, and the first displacement score out of several that are the same.

Train:

Green displacement: [16,8]

Red displacement: [62,33]

Turkmen:

Green displacement: [55,18]

Red displacement: [115,26]

Village:

Green displacement: [65,12]

Red displacement: [137,22]

Emir:

Green displacement: [49,24]

Red displacement: [0,-1131]

The results obtained for this were also unsatisfactory. In this image the alignment was off by a large amount already on the coarsest image, so given that the window sizes decreased going down subsequent levels, the alignment was not corrected.

However those obtained for the same image, but cropping the image by a smaller amount were more satisfactory. It seemed that in this image the border could have helped with the alignment. These were the results obtained for the image with a crop of 50 instead of 200, though for most images, cropping out the border gave better alignment.