**Learning-Based Image Synthesis** **Assignment 1** - Colorizing the Prokudin-Gorskii Photo Collection Overview ============================================================== Prokudin-Gorskii was a visionary who travelled across the Russian empire and clicked three exposures of photographs onto a glass plate using a red, green and blue filter. This was before color photography had been developed. Prokudin-Gorskii is no more but his RGB glass plate negatives has been digitized. We are required to take these glass plate images as input and produce single color images as output using image processing techniques. In other words, the three channels has to be aligned accurately and efficently. Single-scale ============================================================== Naively stacking the three channels one on top of the other won't give a good image. Shown below on the left side is image resulting from simply stacking the three channels. Image alignment needs to be done exhaustively over a search window. A range of [-15,15] is taken and the best image alignment is found using the NCC metric. I tried with both SSD and NCC metric, the results on the low-res image were the same. On high-res images, SSD is faster than NCC but its less accurate. Sum of Squared Differences (SSD) is simply sum(sum((image1-image2).^2)) where the sum is taken over the pixel values. And Normalized Cross-Correlation (NCC) is a dot product between the two normalized vectors - (image1./||image1|| and image2./||image2||). The borders of the images have noise which hurt the results. Hence, I crop 10% on the image on all the sides when checking for alignment.   Image | G offset | R offset -------|------|---------- cathedral | (5, 2) | (12, 3) [Table [states]: Offsets] Multi-scale ============================================================== For high-res images, exhaustive search is time consuming hence a faster search procedure is implemented using image pyramids. The images are represented at multiple scales by taking scale factor as 2 and processing is done sequentially from the smallest image moving on to the largest. I restrict the smallest size not to go below 128x128. The borders of the images have noise which hurt the results. Hence, I crop 20% on the image on all the sides when checking for alignment. Shown below are the results.          Image | G offset | R offset -------|------|---------- cathedral | (5, 2) | (12, 3) harvesters | (58, 16) | (124, 14) icon | (40, 18) | (90, 22) lady | (50, 8) | (108, 12) self_portrait | (78, 28) | (176, 36) three_generations | (50, 14) | (110, 12) train | (42, 6) | (86, 32) turkmen | (56, 20) | (116, 28) village | (64, 12) | (136, 22) [Table [states]: Offsets] Failure case -------------------------------------------------------------- In the case of the Emir of Bukhara, the images do not have the same brightness values. The green channel fails to align properly with blue since green has a different brightness distribution than that of blue and red. Intensity of the pixels in green are significantly different than red and blue. Hence, when we align green with blue and red with blue, there won't be coherence between the two alignments. So when blue and red channels are aligned with green, the alignment happens properly with coherence between the two. A more smarter way to achieve the same would probably be to extract edges out of the images and align the egdes.   Image | B offset | R offset -------|------|---------- emir | (-48, -24) | (56, 18) [Table [states]: Offsets] Own examples ============================================================== These are the results on a few examples of my own choosing from the Prokudin-Gorskii collection.    Image | G offset | R offset -------|------|---------- bridge | (40, 2) | (82, -2) building 1 | (26, 12) | (64, 16) building 2 | (76, 24) | (172, 40) [Table [states]: Offsets] Bells & Whistles ============================================================== Re-implemented the code in pytorch. Script has been submitted in Gradescope.