Assignment #1 - Colorizing the Prokudin-Gorskii Photo Collection
Background
Sergei Mikhailovich Prokudin-Gorskii (1863-1944) [Сергей Михайлович Прокудин-Горский, to his Russian friends] was a man well ahead of his time. Convinced, as early as 1907, that color photography was the wave of the future, he won Tzar’s special permission to travel across the vast Russian Empire and take color photographs of everything he saw including the only color portrait of Leo Tolstoy. And he really photographed everything: people, buildings, landscapes, railroads, bridges… thousands of color pictures! His idea was simple: record three exposures of every scene onto a glass plate using a red, a green and a blue filter. Never mind that there was no way to print color photographs until much later – he envisioned special projectors to be installed in “multimedia” classrooms all across Russia where the children would be able to learn about their vast country. Alas, his plans never materialized: he left Russia in 1918, right after the revolution, never to return again. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress. The LoC has recently digitized the negatives and made them available on-line.
Overview
The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, I extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image. An explanation on how the Library of Congress created the color images on their site is available here.
Approach
Single-scale Alignment
This alignment function takes in two images (and some extra keyword arguments), then tries to align the second image over the first by essentially doing the following procedure:
- Iterate over
[-displacement, displacement]in both i and j axes. I use a displacement of 15 pixels for the whole homework as it works well. - For each iteration, roll the second image using
np.roll, center-crop both images (take 5% off each side), normalize the crops separately, then find the sum of squared distances (SSD) between the two normalized crops. - Return the displacement factors that result in the smallest SSD.
In my experiments, I also tried using normalized cross-correlation (NCC), but it did not work as well as SSD.
Multi-scale Alignment
The multi-scale alignment makes use of the single-scale alignment and calls itself recursively with smaller and smaller versions of the same image. We use a scale factor of 0.5, which means each recursive call the image area gets 4 times smaller (the lengths get 2 times smaller). The recursive call returns a set of displacements, which we scale by the inverse of the scale factor (multiply by 2), and then we pass these initial displacements into our single-scale alignment call, which uses them as the starting point to search over the displacement range (i.e., iterate over [-displacement + initial_displacement, displacement + initial_displacement]), we then just return the result.
The number of times we recurse is determined by a simple formula max(int(floor(log2(min(im.shape)))) - 6, 0), where we use log2 since we use a scale factor of 0.5, and we deduct 6 because we want to have a minimum resolution of the image (I tested and found out that too small images result in bad results, likely due to the loss of important features leading to very bad displacements being propagated and compounded over the scales).
Basic Results
Here are the results after running my multi-scale align function on the images given in the data folder.
Hover over each image to see a brief description and the amount of shift (in pixels) applied to the green and red channels.
Further Results
Here are the results after running my multi-scale align function on the images I selected from the Prokudin-Gorskii Photo Collection.
Analysis
Almost all the images are very well aligned. We can see some alignment issues that cannot be fixed by just translation in the Harversters image for example, where some areas of the image are well aligned but small patches have sudden misalignments, like the chromatically aberrated woman on the left side of the image. One image that is not particularly well aligned is the Lake image, and we will try to fix this in the Better Features section in our Bells and Whistles part.
Bells and Whistles
PyTorch Implementation
I reimplemented the single-scale and multi-scale alignment functions in pure PyTorch (see code submitted). Testing on a few sample images reveals that the implementation is still correct, as the pictures are well-aligned. I will refrain from placing the all the pictures here as they look exactly the same as the ones above, but I have included the Lady and Emir pictures as they are more challenging than others.
Before
After
(PyTorch Implementation)
Automatic Contrasting
I convert the images to the CIELAB color space (also known as the Lab color space), then perform histogram equalization on the Luminance channel. The conversion of color space is needed because applying histogram equalization on RGB channels separately may possibly change the image's color balance, since the relative distributions of the channels would change. Using the Lab color space, we can edit the Luminance of the image without affecting the color balance. We use two different types of histogram equalization, first being my own vanilla histogram equalization implementation in pure numpy, and second being cv2's CLAHE (Contrast Limited Adaptive Histogram Equalization).
Here's some sample images and how they look like before/after the histogram equalization (I chose the images with the poorest contrast):
Before
After
(Own Implementation)
After
(cv2 CLAHE)
Notice that while contrast is definitely improved for most of the images, there are some failure cases. The CLAHE algorithm in general does better at maintaining the image's white balance and overall luminosity due to its adaptivity, and in general the vanilla histogram equalization algorithm tends to darken the image. The darkening is especially noticable in the second last image, and I theorize the darkening effect is due to the white/solid color borders around the image, which is especially large in the second last image. It is possible that we will get much nicer pictures if we combine this step with automatic white balancing, or if we perform border cropping before applying histogram equalization.
Better Features
I use edges (generated from cv2.Canny) to align the images instead of just pixels, and below are sample images showing the result from that:
Before
After
Notice that in both images, the alignment remains roughly the same as before (the shifts change by a little bit, hover over the images to view), and I suspect this may be due to affine/warp transformations (other than translations) being needed to align it. I also tried using image gradients and changing the SSD metric to NCC, but it did not improve the results.