Max Grebinskiy's Home Page for 16-726 Project 1

Project Overview

For this project, the goal was to combine three color channel images from exposures that were taken using red, green, and blue light filters, and produce a single colorized RGB image after properly aligning the three channels. The main technique was using horizontal and vertical translations of the green and red channels relative to the blue channel, and the best alignment was determined by minimizing the SSD of the brightness values of the corresponding pixels.

Initially, alignment was done on the entire image, primarily by brute force iterating through different displacement values and finding which values work best. While this does work, it is very inefficient for higher resolution images. To improve the runtime of this, I implemented an image pyramid at scales of 2, which greatly improved the runtime of aligning each image to be well under a minute.

One issue I ran into was proper alignment for the picture of Emir of Bukhara. To try and resolve this, another approach I looked at was using NCC as a loss metric. This approach ended up producing the same results as SSD (± 1 pixel in each direction) for all images, but runs significantly slower. This approach also failed to properly align the Emir. Because of this, I chose to omit NCC in my functions. This image primarily fails because its channels do not actually contain the same brightness values, so individual pixel comparisons based on brightness are unlikely to work. One potential idea is to use convolutions to parse the horizontal and vertical edges of the Emir, and try aligning that instead.

One "bells and whistles" operation made was cropping black and white borders off images. This was done by setting a threshold on the brightness values on each side of the image over the average brightness value of that row/column. While this does have an effect in cropping the sides, finding the correct threshold is tricky. When I slightly tightened my brightness thresholds, this caused a large chunk of the top of the train image to be cropped off as a result (due to it having an overly exposed skyline). Differentiating between overexposure and white border (or underexposure and black borders) is a broader challenge to tackle, and this approach may not be the best for that.