Following steps were used to generate colorized images:

1. Exhaustively search over a window
2. Option to normalize the image to handle different image intensities
3. Scoring methods: Sum of Squared Differences (SSD) and Normalized cross-correlation (NCC)
4. Image Pyramid

One shot results using ssd and ncc with channel displacement in the bottom:

Source	Normalized SSD	NCC
Channel Displacements:	(-152, -240) and (144, 240)	(-120, -120) and (120, 120)
Channel Displacements:	(-2, -5) and (1, 7)	(-2, -5) and (1, 7)
Channel Displacements:	(-168, -240) and (192, 240)	(-64, -120) and (-120, 120)
Channel Displacements:	(-72, -240) and (8, 240)	(-120, -120) and (40, 120)
Channel Displacements:	(-136, -240) (-8, 240)	(-120, -120) and (-8, 120)
Channel Displacements:	(-136, -240) (40, 240)	(-120, -120) and (40, 120)
Channel Displacements:	(-144, -240) (40, 240)	(-120, 120) and (24, 120)
Channel Displacements:	(240, 0) and (80, 240)	(120, 0) and (-120, 120)
Channel Displacements:	(-104, -240) and (-8, 240)	(-96, -120) and (-16, 120)

Pyramid based method:

Some images are blurred because the window sized is limited to [-40,40] due to computation

Source	Normalized SSD	NCC
Channel Displacements:	(-72, -120) and (192, 120)	(120, 120) and (120, -120)
Source	Normalized SSD	NCC
Channel Displacements:	(-24, -48) and (40, 40)	(-20, -32) and (20, 20)
Channel Displacements:	(-2, -5) and (8, 40)	(-2, -5) and (4, 20)
Channel Displacements:	(-5, -42) and (40, 40)	(-6, -32) and (20, 20)
Channel Displacements:	(-17, -48) and (0, 40)	(-17, -32) and (0, 20)
Channel Displacements:	(-17, -41) and (40, 40)	(-17, -32) and (20, 20)
Channel Displacements:	(-13, -20) (5, 20)	(-18, -32) and (4, 20)
Channel Displacements:	(-15, -15) and (5, 15)	(-15, -15) and (12, 15))
Channel Displacements:	(30, 0) and (10, 30)	(15, 0) and (-15, 15)