16726-Learning Based Image Synthesis-Spring 24

Assignment #1 - Colorizing the Prokudin-Gorskii Photo Collection

Student Name: Peipei Zhong, Andrew ID: peipeiz

Overview

The assignment involves processing digitized Prokudin-Gorskii glass plate images to create a color image with minimal artifacts. We need to separate the three color channels and align them into a single RGB image. The task requires efficient alignment techniques due to the large size of the original images. Starter Python code is available but not mandatory.

Simple-Scale Search

The easiest way to align the parts is to exhaustively search over a window of possible displacements, and for small images(eg. Cathedral.jpg), [-15, 15] is enough. So I applied 15 as the width and height of the displacement window.

I tried both ssd and ncc metrcs.

• Sum of Squared Differences (SSD): (sum((image1-image2).^2))

  • smaller SSD means better algnment

• Normalized cross-correlation (NCC): first normalized image1 and image2, then apply dot product (image1/||image1||) * (image2/||image2||)

  • NCC values range from -1 to 1, where 1 means exactly the same, -1 means exactly the opposite, and 0 means no correlation

Their difference is that, NCC is not sensitive to linear changes in brightness, and can deal with light changes well.

The optimizations I implemented:

• Crop: I cropped the image to the same proportions on all sides and used the middle part for calculations. This is because the middle part is the main body of the image, with more details and changes, while the edges often have little change and are difficult to align, resulting in larger errors. Doing so also reduces image processing time.

SSD	NCC
Offsets: g: (5,2) r:(12, 3)	Offsets: g: (5,2) r:(12, 3)

Image Pyramid

For larger images, we need to enlarge the displacement search window, and the search time would be really long(my computer doesn't even run). In this case, I applied image pyramid method.

1, Use transform.pyramid_gaussian to get pyramid images;

2, I didn't use all levels of pyramid, I only used 512*512 or even larger images;

3, Applied alignment to low level images, and then applied the displacement obtained in this layer to the next layer

4, The highest layer is the original image, aftering aligning it, I got the final displacement

emir:

SSD	NCC
Offsets: g: [49, 24] r: [0, -196]	Offsets: g: [49, 24] r:[0, -196]

harvesters

SSD	NCC
Offsets: g: (59, 17) r:(123, 14)	Offsets: g: (59, 17) r:(123, 14)

icon

SSD	NCC
Offsets: g: (40, 17) r:(89, 23)	Offsets: g: (40, 17) r:(89, 23)

lady

SSD	NCC
Offsets: g: (49, 8) r:(109, 11)	Offsets: g: (49, 8) r:(109, 11)

self_portrait

SSD	NCC
Offsets: g: (78, 29) r:(176, 37)	Offsets: g: (78, 29) r:(176, 37)

three_generations

SSD	NCC
Offsets: g: (50, 14) r:(110, 12)	Offsets: g: (50, 14) r:(110, 12)

train

SSD	NCC
Offsets: g: (42, 6) r:(87, 32)	Offsets: g: (42, 6) r:(87, 32)

turkmen

SSD	NCC
Offsets: g: (55, 20) r:(116, 28)	Offsets: g: (55, 20) r:(116, 28)

village

SSD	NCC
Offsets: g: (64, 12) r:(137, 22)	Offsets: g: (64, 12) r:(137, 22)

a few images chosen from the original collection

piony

SSD	NCC
Offsets: g: (51,3) r:(105,-5)	Offsets: g: (51,3) r:(105,-5)

man using tool

SSD	NCC
Offsets: g: (56,14) r:(123,14)	Offsets: g: (56,14) r:(123,14)

Extra Credit

1, Gradient

Instead of aligning based on RGB similarity, try using gradients.

xxxxxxxxxx
def compute_gradient(image):
    dx = sobel(image, axis=0)  # x-axis
    dy = sobel(image, axis=1)  # y-axis
    gradient = np.sqrt(dx**2 + dy**2)  
    return gradient

Offsets: g:[53, 13], r: [111, 9]

This time, emir.tif have much better result! I think the reason may be that there is a lot of similar space(like the wall) in the background of this picture, which will interfere with the minimum value of the SSD. At this time, using image features such as edges (gradient is an expression of edges) will give better results.

Offsets: g:[49, 24], r: [107, 40]

Offsets: g:[60, 18], r: [123, 14]

Offsets: g:[64, 11], r: [137, 21]

Discussion

1, I got totally same results of SSD and NCC, since NCC is not sensitive to linear changes in brightness and can deal with light changes well, maybe the light changes in these images is not significantly.

2, My methods have good performances to thses images instead of emir.tif at first, so I tried to use gradient instead of pixels to calculate the metrics, and it gets better. But the effect improvement is not obvious on other images