**16-726
Collorizing the Prokudin-Gorskii photo collection** **Anish Jain (anishaja)** Overview =============================================================================== The goal of this project is to colorize the Prokudin-Gorskii photo collection. The collection contains a series of glass plate negatives taken by Russian photographer Sergei Mikhailovich Prokudin-Gorskii in the early 20th century. The collection is housed at the Library of Congress. The photos are in black and white and were taken using a special camera that took three black and white photos in quick succession, each with a different color filter (red, green, and blue). The goal of this project is to colorize these photos by aligning the three black and white photos and then combining them to create a color photo. The project will use a combination of image processing techniques and machine learning to achieve this goal. The naive solution is to seach over a window of possible shifts and find the shift that minimizes the difference between the three images. This is a naive solution because it is computationally expensive. I experimented with both SSD and NCC loss. Another idea that helped speed up the process was to find the alignment by using **center crop** of the images. This is because the borders of the images are often not aligned, and have problematic artifacts. After getting reasonable results from the naive solution, I extended the algorithm to use **coarse-to-fine alignment**. This is a common technique in computer vision and image processing. The idea is to first align the images at a low resolution and then refine the alignment at a higher resolution. This is a good idea because the alignment is more robust at lower resolutions and the computation is faster. I limit the **height of the pyramid to be 7**. To build the pyramid, the images are downsampled by a factor of 2 at each level as we go up the pyramid. The alignment is then refined at each level of the pyramid. The final alignment is used to colorize the images. This worked for all the images except the **Emir of Bukhara**. The problem with this image is that the three images are not aligned properly. This is because the **channels do not have the same brightness**. To solve, I extended my algorithm to use **edges to align the images**. This is a good idea because the edges are more robust to brightness changes. The edges are found by applying **gaussian blur** to the images and then subtracting the blurred image from the original image. The edges are then used to align the images. **This solved the mystery of the Emir of Bukhara!** Results =============================================================================== Example images ------------------------------------------------------------------------------- Below are the results by using my final algorithm - pyramid and edge alignment. **After the pyramid and edge alignment**, it was observed that **both NCC and SSD gave exactly the same shifts**. Hence, only the shifts for SSD have been provided below. The shifts are in the format (x, y) where x is the shift in the x direction and y is the shift in the y direction. The shifts are provided for green-blue, followed by red-blue. | Name | Image | Shift | |-------------------|-------------------------------------------------------------------------------------|---------------------| | Cathedral | | (5,2), (12, 3) | | Emir of Bukhara | | (49,24), (106,41) | | Turkmen | | (57,22), (117, 29) | | Harvesters | | (60, 18), (124, 11) | | Icon | | (39, 16), (89, 23) | | Three Generations | | (53,12), (113, 10) | | Lady | | (56, 9), (120, 13) | | Self portrait | | (77, 29), (175, 37) | | Village | | (64, 10), (137, 21) | | Train | | (52, 6), (85, 29) | Experimentation on my own images ------------------------------------------------------------------------------- I also experimented with my own images. The images are taken from the internet and are in black and white. The images are of the same scene but taken with different color filters. The images are aligned and then combined to create a color photo. The results are below. | Name | Image | Shift | |------------|----------------------------------------------------------------------------------------|-----------------------| | Siren | | (50, -5), (97, -24) | | Suna river | | (25, -5), (102, -12) | Bells and Whistles =============================================================================== Pytorch Implementation ------------------------------------------------------------------------------- I implemented my ideas as described in the Overview section using PyTorch. I used the following PyTorch functions: torch.roll, torch.mean, torch.std, torch.linalg.norm, torch.sum etc. The results are the same. However, the PyTorch implementation is faster than the numpy implementation. The code is submitted in the file **helper_pytorch.py**. Auto white balance ------------------------------------------------------------------------------- I implemented auto white balance to improve the colorization of the images. The idea is to **find the average color of the image and then scale the color channels so that the average color is white**. This is a good idea because the average color of the image is often not white. The average color is found by taking the mean of the color channels. The color channels are then scaled so that the average color is white. The results are below. | Original | Auto white balance | |-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------| | | | Auto Contrast ------------------------------------------------------------------------------- I implemented auto contrast to improve the colorization of the images. The idea is to **find the minimum and maximum values of the image and then scale the values so that the minimum value is 0 and the maximum value is 255**. This is a good idea because the minimum and maximum values of the image are often not 0 and 255. The minimum and maximum values are found by taking the minimum and maximum of the color channels. The color channels are then scaled so that the minimum value is 0 and the maximum value is 255. The results are below. | Original | Auto contrast | |------------------------------------------------------------------------|--------------------------------------------------------------------------------------| | | | Better features : Using edges for alignment ------------------------------------------------------------------------------- I implemented a better feature for alignment. The idea is to use the edges of the images to align the images. This is a good idea because the edges are more robust to brightness changes. The edges are found by applying **gaussian blur** to the images and then subtracting the blurred image from the original image. The edges are then used to align the images. This worked the best for the Emir of Bukhara since the channels do not have the same brightness. The results are below. | Original | Edge alignment | |-----------------------------------------------------------------------------------|------------------------------------------------------------------------| | | | Auto Cropping ------------------------------------------------------------------------------- I implemented auto cropping to improve the colorization of the images. I did this by identifying rows and columns where more than 70 percent of the pixels have values outside of the middle 70% of the image pixels. I then cropped the image to remove these rows and columns. The results are below. | Original | Auto cropping | |---------------------------------------------------------------------------|-------------------------------------------------------------------------------------| | | | | | |