16-726 Learning-Based Image Synthesis

Spring 2024

Assignment #1 - Colorizing the Prokudin-Gorskii Photo Collection

Yifei Liu

I. Introduction

For the first assignment, we take the digitized Prokudin-Gorskii glass plate images (in seperate color channels) and applied image processing and alignment techniques to produce a single RGB color image. Several high-resolution and low-resolution images are provided for experiments.

There are 2 main tasks:

Single-scale alignment

Corase-to-fine pyramid alignment

II. Results

Single-scale alignment

This task involves searching over a pre-defined window of possible displacements, scoring each using the below metrics, taking the displacement with best score, and aligning the images with the displacement. I implemented three metrics:

SSD: single l2 norm between the images

NCC: slower than SSD but invariant to local average intensity and contrast

SSIM: more robust to changes in brightness and contrast, making it a better fit for perceptual similarity as the glass plate have various brightness

The aligned low-resolution image cathedral.jpg and the red and green displacements are displayed below.

Pyramid alignment

To handle the large .tif images, I implemented image pyramid, a faster search procedure. An pyramid represents image at multiple scales and the processing is done sequentially starting from the top of the pyramid (smallest image). After experimenting with different metrics and parameters, SSD on edges produce the best results across all images. (See Section III.3 for comparision of aligning in RGB). The aligned images (both low and high resolution) and the corresponding red and green displacement are displayed below. The process take ~50 sec to complete.

Own images

Here are the alignment results of several low and high resolution images I chose from the LoC digitalized collection.

Low Resolution

High Resolution

III. Bells & Whistles

1. Automatic cropping

Since all the images have white and black border, I implemented an automatic cropping techniques which search over the 20% image dimention from the margin for significant intensity changes and crop the images. There are marks and noise on the border so a single mask for black and white pixel won't work well.

Here's the original lady.tif and after cropping and the pyramid alignment.

2. Automatic contrasting

The 3 channels in glass plate images often differ in brightness. Though SSIM mitigate the problem, it is not tolerable to dramatic difference as shown in below image. To address this, I normalize the intensity of the channel images to have the same mean and standard deviation. This help to bridge the contrast gap, but still can't ensure the images have exact same contrast.

3. Better features

Given the contrast difference even after automatic contrasting and the various problems with measuring images with RGB similarity, I also tried to align based on edges, which works much better and faster than the other RGB metrics. The shifted image and reference are measured using SSD on canny edges. As shown below, Emir that is hard to align in RGB is able to align well.