16-726: Learning-Based Image Synthesis - Assignment 1 - prabhdes

Overview:

The major task of this image was to align images in RGB spaces to a single color image. There are multiple ways we can go about it but here we start of with the simiplest thing. We take a patch in source image and we try to find a similar patch of image in the target image. To calculate this similarity we can do multiple different metric like SSD, NCC, SSIM, etc. I personally experimented with NCC and SSD. I found SSD to be faster but NCC to be more accurate as it is more invariant to brightness. Once you find a match you basically have to check where in the source image did you found your target patch and just add that displacement and ideally that should work.

However, we don't live in an ideal world and if you iterate a patch over complete image that takes a lot of time. To resolve this I have basically taken the center patch (because corners are irregular accross images) and have only searched for a fraction of center image in target image. We know that the images are disalligned but we also know that ther can't be adrastically huge displacement and I took benefit of that fact and only searched over 1/8th of the center patch in target image. I also added the functionality to search for multiple patches as the center most patch might not have the best features. There are also much more robust feature extraction techniques like SIFT, SURF or even other feature descriptors in OpenCV that could be used and one can also manually select same points in different images and the distance between them and that should work however I stuck to NCC for most part in this assignment.

Finally this worked for all images except emir.tif and three_generations.tif and the three_generations.tif got resolved when I implemented the gaussian pyramid. In the Gaussian Pyramid approach we basically apply gaussian filter of increasing size (double size than last level) and subsample the image based on desired level to reduce the aliasing effects. This way when we apply our alignment technique over each image (smaller to bigger) we will get incrementally correct locations and that would finally lead to the final displacement we are looking for. This again however did not work and I was hence not able to align emir.tif.

PS: I wanted to try homography and edge detection but wasn't able to in the given time.