Assignment 1¶
- Sihan Liu (sihanliu@andrew.cmu.edu)
- In this homework, I implemented the Sum of Squared Differences (SSD) and the normalized cross-correlation (NCC) to evaluate the degree of alignment between images. To accelerate the process for high-resolution images, I integrated a multi-scale search strategy using an image pyramid approach. For extra credits, I did 1. Pytorch-based implementation 2. automated image cropping 3. contrast adjustment 4. white balance
1. Single-Scale Alignment¶
Cathedral w/ SSD (Green Displacement: 5, 2; Red Displacement: 12, 3):
Cathedral w/ NCC (Green Displacement: 5, 2; Red Displacement: 12, 3):
2. Multi-Scale Pyramid Alignment¶
Default Settings:
- crop ratio: 0.25
- metric: ssd / ncc
- output image quality: 60
Emir ssd (Green Displacement: 49, 24; Red Displacement: 103, 61):
Emir ncc (Green Displacement: 49, 24; Red Displacement: 105, 58):
Harvesters ssd (Green Displacement: 58, 16; Red Displacement: 120, 13):
Harvesters ncc (Green Displacement: 58, 16; Red Displacement: 120, 13):
Icon ssd (Green Displacement: 41, 16; Red Displacement: 89, 23):
Icon ncc (Green Displacement: 41, 16; Red Displacement: 89, 23):
Lady ssd (Green Displacement: 48, 8; Red Displacement: 88, 12):
Lady ncc (Green Displacement: 48, 8; Red Displacement: 88, 12):
Self_portrait ssd (Green Displacement: 72, 27; Red Displacement: 160, 33):
Self_portrait ncc (Green Displacement: 72, 27; Red Displacement: 162, 33):
Three_generations ssd (Green Displacement: 40, 13; Red Displacement: 96, 13):
Three_generations ncc (Green Displacement: 40, 13; Red Displacement: 96, 13):
Train ssd (Green Displacement: 32, 5; Red Displacement: 72, 32):
Train ncc (Green Displacement: 32, 5; Red Displacement: 71, 32):
Turkmen ssd (Green Displacement: 48, 8; Red Displacement: 88, 12):
Turkmen ncc (Green Displacement: 48, 8; Red Displacement: 88, 12):
Village ssd (Green Displacement: 65, 12; Red Displacement: 138, 22):
Village ncc (Green Displacement: 65, 12; Red Displacement: 138, 22):
images from Prokudin-Gorskii collection¶
image1 ssd (Green Displacement: 8, 16; Red Displacement: 16, 24):
image1 ncc (Green Displacement: 8, 16; Red Displacement: 16, 24):
image2 ssd (Green Displacement: 24, 6; Red Displacement: 58, 5):
image2 ncc (Green Displacement: 24, 6; Red Displacement: 58, 5):
(Extra Credit) Bells & Whistles¶
1. Pytorch Version Alignment:¶
def cal_ssd_torch(a, b):
ssd = torch.sum((a - b) ** 2)
return ssd
def cal_ncc_torch(a, b):
norm_a = (a / torch.norm(a)).flatten()
norm_b = (b / torch.norm(b)).flatten()
ncc = norm_a @ norm_b
return ncc
def align_torch(a, b, metric='ssd', min_bound=-15, max_bound=15):
best_score = None
best_xy = None
metric_func = cal_ssd_torch if metric == 'ssd' else cal_ncc_torch
b = crop(b)
for x_dis in range(min_bound, max_bound):
for y_dis in range(min_bound, max_bound):
a_xy = torch.roll(a, (x_dis, y_dis), (0, 1))
a_xy = crop(a_xy)
score_xy = metric_func(a_xy, b)
if best_xy is None or \
(metric == 'ssd' and score_xy < best_score) or (metric == 'ncc' and score_xy > best_score):
best_score = score_xy
best_xy = (x_dis, y_dis)
print("Best Displacements: " + str(best_xy))
best_a = torch.roll(a, best_xy, (0, 1))
return best_a, best_xy
def _align_pyramid_torch(a, b, metric, x_bound, y_bound):
best_score = None
best_xy = None
metric_func = cal_ssd if metric == 'ssd' else cal_ncc
a, b = crop(a), crop(b)
for x_dis in range(x_bound[0], x_bound[1]):
for y_dis in range(y_bound[0], y_bound[1]):
a_xy = torch.roll(a, (x_dis, y_dis), (0, 1))
score_xy = metric_func(a_xy, b)
if best_xy is None or \
(metric == 'ssd' and score_xy < best_score) or (metric == 'ncc' and score_xy > best_score):
best_score = score_xy
best_xy = (x_dis, y_dis)
best_a = torch.roll(a, best_xy, (0, 1))
return best_a, best_xy
def align_pyramid_torch(a, b, metric='ssd'):
if max(a.shape + b.shape) <= 64:
range_dis = 15
center = (0, 0)
x_bound = -(range_dis + 1) + center[0], (range_dis + 1) + center[0]
y_bound = -(range_dis + 1) + center[1], (range_dis + 1) + center[1]
_, xy_dis = _align_pyramid_torch(a, b, metric, x_bound, y_bound)
return xy_dis, range_dis, center
else:
a_r = torchvision.transforms.Resize((int(a.shape[0] * .5), int(a.shape[1] * .5)))(a)
b_r = torchvision.transforms.Resize((int(b.shape[0] * .5), int(b.shape[1] * .5)))(b)
xy_dis, range_dis, center = align_pyramid_torch(a_r, b_r)
center = (center[0] + xy_dis[0], center[1] + xy_dis[1])
range_dis = range_dis // 2
range_dis = 1 if range_dis <= 1 else range_dis
x_bound = -(range_dis + 1) + center[0], (range_dis + 1) + center[0]
y_bound = -(range_dis + 1) + center[1], (range_dis + 1) + center[1]
_, xy_dis = _align_pyramid_torch(a, b, metric, x_bound, y_bound)
return xy_dis, range_dis, center
2. Auto cropping:¶
step1: convert the image to gray scale images step2: use Canny Edge Detection step3: dilate the image to get the boundary information step4: remove the boundary
w/o cropping:
w/ cropping:
2. Auto contrast:¶
step1: convert the RGB image to the LAB image format setp2: applied cv2.createCLAHE to the L channel w/o auto contrast:
w/ auto contrast:
