Assignment #1 - Colorizing the Prokudin-Gorskii Photo Collection

2.1 Single-scale Implementation (LOW Resolution)

The blue channel serves as the reference layer against which the green and red channels are matched. In the single-scale implementation for low-resolution images, a window size of 15x15 pixels is utilized. The algorithm iteratively searches for the optimal alignment by looping through displacement pairs within a range of -15 to 15 pixels in both the row and column dimensions, selecting the pair that yields the best metric match.

Sum of Squared Differences (SSD) is one metric to assess the disparity between two images. It calculates the sum of the squared intensity differences between corresponding pixels. The SSD formula is given by:

SSD(I₁, I₂) = Σ (I₁(x, y) - I₂(x, y))²

where I₁ and I₂ are the two images being compared, and the sum is computed over all pixels at position (x, y).

Normalized Cross-Correlation (NCC) is another metric for measuring the similarity between two images. It is calculated as the sum of the product of corresponding pixel intensities, normalized by the product of their standard deviations. The NCC formula is:

NCC(I₁, I₂) = (Σ (I₁(x, y) - µ₁) * (I₂(x, y) - µ₂)) / √(Σ (I₁(x, y) - µ₁)² * Σ (I₂(x, y) - µ₂)²)

Here, I₁ and I₂ represent the two images, µ₁ and µ₂ are their mean values, and the sums are over all the pixel positions (x, y).

Both normalized cross-correlation (NCC) and sum of squared differences (SSD) metrics were experimented for performance. While SSD proved to be the faster of the two, there was no significant difference in the quality of the results when compared to NCC. So I went with SSD for the metric.

Problem encountered: as is shown in the left picture, the blue channel lies above the other two channels consistently at first. I suspected that the issue was related to border tampering as the window size search was exhaustive and shouldn't have caused the mismatch by itself. I applied cropping and voila, the cathedral gets perfectly aligned as is shown on the right.

2.2 Coarse-to-Fine Pyramid (HIGH Resolution)

Since cropping proved to improve performance in the single-scale cases, I also incorporate it into the coarse-to-fine implementation. Since in higher resolution, it calls for better precision, so I experimented a bit and found a more precise cropping strategy. It starts with cropping the image into three equal sections by height. The dimensions for cropping are calculated as a percentage of the image's width and height, specifically 8% of the width and 2.5% of the height. Say, the total height of the image is total_height, I get height = total_height // 3. Then, to account for the extra white space on the top and at the bottom, I use "[3 * 2.5% of the height : height - 2.5% of the height]" to get the first (blue) channel. This cropping technique is essential to remove the extra space around the original plate images and is also crucial for replicating the final displacement results.

Following the cropping, a pyramid search algorithm is implemented recursively with three layers. The process starts by resizing the image by a factor of 2 at each level of recursion until the smallest size is reached. Then displacements from the lower levels are scaled up and added to the current level's calculations. At the lowest level, displacement is (0, 0). The rolled image then gets aligned for this layer using the SSD metric, with the sum of displacements providing the final value, which is carried up through the layers. The hyperparameters for the search windows are set larger for the bottom levels and smaller for higher levels to optimize processing time for larger-pixel images. In my implementation, the bottom layer uses a window of 5(Left-Right) by 32(Up-Down), the middle layer 4 by 8, and the top layer 3 by 6. I also leveraged the characteristic that the given images are better horizontally aligned than vertically aligned to reduce search space. This hyperparameter choice significantly reduces the search time, particularly in the non-PyTorch implementation, which completes in under 25 seconds for each large TIFF image, with even faster performance noted in PyTorch versions.

Problem Encountered: one issue encountered was the differing methods by which various image processing libraries interpret image data. For instance, skimage (scikit-image) typically handles images as floating-point numbers normalized between 0 and 1 when images are read using its functions. On the other hand, OpenCV reads images directly as integers ranging from 0 to 255, which corresponds to the standard 8-bit format per channel. Moreover, OpenCV often defaults to BGR (Blue, Green, Red) channel order rather than the conventional RGB (Red, Green, Blue). Conversely, PIL (Python Imaging Library) and its fork, Pillow, also read images into integers ranging from 0 to 255 but maintain the standard RGB order. Navigating these inconsistencies required careful attention to ensure that images were properly processed and saved across the various libraries.

Problem encountered: another small problem encountered was the requirement for channels to have identical shapes for SSD/NCC matching. Since the total image height was not always perfectly divisible by three, I had to slightly crop sizes (1-2 pixels) to ensure uniformity across all three channels when total height can not be divided by three.

Problem encountered: additionally, the displacement values needed to be scaled according to the size reduction factor. For example, a displacement of 3 pixels in a downsized image of 10 by 10 pixels corresponds to a 6-pixel displacement in the original size of 20 by 20 pixels. While this scaling turns out to be convenient, it required careful consideration to determine the appropriate factors that matter for accurate image alignment.

2.3 PyTorch Implementation: Single-scale & Multiscale Pyramid

For this bonus Bells and Whistles, I implemented the torch equivalent of the previous implementation for both single-scale and multi-scale case.

The optimized torch implementation manages to complete a large TIFF image within less than 10 seconds.

Problem encountered: initially, the refactoring from the non-Torch version to the Torch version resulted in significant displacement. I managed to trace back the issue to the F.interpolate function, which didn't yield as good results as sk_resize in skio, probably due to anti-aliasing. To overcome this, I converted the images back to numpy arrays for resizing using sk_resize, then converted them back to torch vectors for subsequent steps. This solution was able to replicatd the results of the non-Torch implementation perfectly.

2.4 PyTorch Implementation with Edge Feature

Building upon the Torch version, I changed the feature to be edge-based. I expected edge detection to yield better results than plain RGB values because edge detection focuses on structural information and contours within an image, which can provide a more robust basis for image alignment and comparison. I computed the edge for each channel, then used SSD on the edge feature. This approach solved the alignment issue with the 'emir' image.

2.5 PyTorch Implementation with Edge Feature and Automatic Contrast Adjustment

For the final bonus implementation, I experimented with automatic contrasting using three different methods. The above results from left to right are respectively autocontrast with cutoff 0.1, autoenhance with gamma 1.5, and Lookup Table (LUT) with a mapping that makes darker area darker and brighter area not adjusted.

The autocontrast function is a tool that automatically adjusts the contrast of an image by remapping the lightest and darkest pixels to white and black, respectively. This remapping stretches the intermediate pixel values, enhancing the overall contrast. The cutoff parameter allows for discarding a certain percentage of the lightest and darkest pixels, which is useful for ignoring outliers that might skew the contrast adjustment. Essentially, this method redistributes the image's histogram of pixel values to utilize the full tonal range of the display.

Autoenhance, specifically when enhancing brightness, employs a gamma correction technique. The gamma value alters the luminance of the image; a gamma value greater than 1 makes the image appear darker, while a value less than 1 makes it lighter. The operation is nonlinear and adjusts the midtones while preserving the black and white points, thus impacting the image's perceived brightness without significantly changing its contrast or color hues. It's particularly useful for correcting the exposure of an image.

Lookup Table (LUT) curve adjustment is a powerful technique where a predetermined set of values (the LUT) is used to remap the original pixel values of an image to new ones, based on a curve function. This can be used to create an S-shaped curve which typically enhances contrast. Dark tones become darker, and bright tones become brighter, while mid-tones remain relatively unaffected. This selective adjustment is akin to dodging and burning techniques in traditional photography, and is often used to add depth and dimensionality to an image. The S-curve is particularly effective because it can add contrast while maintaining a natural look, as opposed to a simple linear increase in contrast which can sometimes give an image an artificial appearance.

In the Harvesters image example, the original picture looks good, but a little pale. The autocontrast one and the LUT table one indeed make the picture more contrast and therefore more vivid, while as the original image is already pretty bright, so it does not look good for the autoenhanced one in the middle.