_________________________________________________________

HW1

In this homework, I explore a task that takes us back to the 19th century, involving the work of Sergei Mikhailovich Prokudin-Gorskii. He developed a technique for capturing color photographs of real-world scenes by recording three separate exposures on a glass plate, each through a red, green, or blue filter. However, directly overlaying these three images without adjustment is impractical due to the affine transformations that differ among the image channels. My objective is to align these images to create a composite RGB image that achieves both high image reconstruction quality and computational efficiency for processing large-scale images.

In the subsequent sections, I will outline my methodology, starting with "single alignment" strategies for low-resolution images, followed by the "pyramid alignment" technique for handling high-resolution images. The core components of the project are implemented in PyTorch, with the exception of input/output operations. Additionally, an ablation study will be conducted to assess the impact of various metrics used for measuring image similarity.

Method

Single Alignment

To align three image channels, I begin by anchoring one image in a channel and determining the translational adjustments needed for the other two channel images. By setting one image as a reference, I explore a defined movement range (e.g., [-15, 15]) for the second and third images. For each potential shift, I compute a score to assess the similarity between the reference image and the adjusted image. After evaluating all possible shifts within the specified range, I select the shift that yields the highest similarity score, thereby aligning the three images.

A key detail in my approach involves addressing the boundary artifacts introduced by older camera equipment in each channel image. These boundaries can skew the similarity calculations between images. To mitigate this, I exclude a small portion of the image boundaries from consideration.

To validate my "Single Alignment" technique, I experimented with "cathedral.jpg." The unaligned image, when the three channels are simply merged, shows clear pixel misalignments. In contrast, applying the Single Alignment method with the Sum of Squared Differences (SSD) metric significantly improves alignment. The adjustments for the green channel image are dh = 5 and dw = 2, and for the red channel image, dh = 12 and dw = 3, where "dh" and "dw" represent vertical and horizontal shifts, respectively.

While the method proves effective, it faces a notable limitation due to the constrained search range, such as the [-15, 15] window initially set. This approach becomes impractical for larger displacements, for instance, if we aim to explore within a [-200, 200] range. Employing a brute force search over such an expansive range significantly increases computational complexity, making it nearly impossible to apply on large-scale images. In the subsequent section, I will address and overcome this challenge.

Pyramid Alignment

To address the previously mentioned challenge, I introduced a method known as "Pyramid Alignment," leveraging the concept of an image pyramid, as depicted below. My approach unfolds in the following steps:

I constructed an image pyramid with four levels, scaling the images to [0.125, 0.25, 0.5, 1] relative to the original image size. The process begins at the topmost level of the pyramid, which, in our case, is the image scaled to 0.125 times the original size. Additionally, I defined a search window for each level; in my experiments, setting the search window to [20, 5, 5, 5] proved to be a balanced choice for achieving reasonable results while maintaining computational efficiency.

2. In each level i, I use my Single Alignment method, by searing in the defined window, I find the best translation offsets (dh_i, dw_i).

3. Then I past my offsite (dh_i, dw_i) to the next level i+1 in the pyramid, the next level starts from the offset (2*dh_i, 2*dw__i).

4. This process—repeating steps 2 and 3—culminates in a final solution for the alignment.

Employing this pyramid strategy, I was able to align high-resolution images with high quality in under a minute. Qualitative results of this methodology are presented in the ablation study section.

Metrics

I employed three distinct metrics to achieve image alignment:

Sum of Squared Differences

Normalized Correlation

Structural Similarity Index Measure

The results of using different metrics are shown in the ablation study. But mentioned before, the computational complexity is: SSIM > NCC > SSD.

Ablation Study

The table below presents the alignment outcomes in comparison to a baseline, including the specific offset values.

Baseline	SSD	NCC	SSIM

The results highlighted above reveal certain limitations when applying the Sum of Squared Differences (SSD) and Normalized Cross-Correlation (NCC) metrics, as demonstrated by the examples of self_portrait.png and emir.png for both SSD and NCC, and the village.png example specifically for NCC. Although these issues can be addressed by employing the Structural Similarity Index (SSIM), it's important to note that there is a trade-off in computational complexity among these metrics.

Filename	SSD	NCC	SSIM
emir.tif	Green:49,24 Red:103,61	Green: 49,24 Red:105,58	Green:50,23 Red:106,41
harvesters.tif	Green:60,17 Red:124,13	Green: 60,17 Red:125,14	Green:59,16 Red:123,13
icon.tif	Green:41,17 Red:90,23	Green: 41,17 Red:90,23	Green:40,17 Red:89,23
lady.tif	Green:56,9 Red:116,12	Green: 55,10 Red:118,13	Green:56,9 Red:120,12
self_portrait.tif	Green:79,29 Red:155,34	Green: 79,29 Red:155,34	Blue:-78,-29 Red:98,8
three_generations.tif	Green:53,14 Red:112,11	Green: 54,15 Red:112,12	Green:56,16 Red:113,10
train.tif	Green:43,6 Red:87,32	Green: 44,7 Red:88,33	Green:41,7 Red:85,31
turkmen.tif	Green:56,21 Red:116,28	Green: 56,21 Red:116,29	Green:57,21 Red:116,29
village.tif	Green:65,12 Red:138,22	Green: 155,155 Red:155,-155	Green:66,12 Red:138,22

References

Image Pyramids Figure