Assignment 2

Brief Overview

Goal of this assignment is to seamlessly blend an object or texture from a source image into a target image. The most naive method would be to copy and paste pixels of source image to the target, however this approach performs poorly, leaving noticable seams even for the cases where background colors are the same.

How do we get rid of these noticable seams? If we can somehow smoothen the border while pasting source image in the target image, the blending would look much smoother compared to the naive method above. This is possible through gradient-based fusion, where we smooth out the border by adjusting the gradients of the source image and target images. We can formulate this as a least-square equation as follows:

$$ \mathbf{v} = \text{arg} \min_v \sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - (s_i-s_j))^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - (s_i-s_j))^2 $$

where,
S: Source Region, i: point in S; j: neighbor of i; s: pixel intensity of source image

v: new intensity values we need to find for blending;

t: pixel intensity of target image.

The first term of the equation essentially reconstructs the source image into the target image. The second term of the equation tries to blend the rough edges of the source image with the target image, by preserving the local gradient using $t_j$. These equations can be solved using a least-square solver.

This method is called Poisson Blending and is one of a kind in gardient-based fusion methods.

Another kind of this method is Mixed Blending. This method is similar to Poisson blending, with a small difference that instead of optimising least-square gradient wrt to source image, we instead set it as maximum of the target and source gradients with $d_{ij} = max(s_{i} - s_{j}, t_{i} - t_{j})$

$$ \mathbf{v} = \text{arg} \min_v \sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - d_{ij})^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - d_{ij})^2 $$

Mixed blending is helpful when we want to blend backgrounds, which we will show in example below.

Toy Problem

Denote the intensity of the source image at (x, y) as s(x,y) and the values of the image to solve for as v(x,y). For each pixel, then, we have two objectives:

1. Minimize $(( v(x+1,y)-v(x,y)) - (s(x+1,y)-s(x,y)) )^2$, so the x-gradients of v should closely match the x-gradients of s.
2. Minimize $(( v(x,y+1)-v(x,y)) - (s(x,y+1)-s(x,y)) )^2$, so the y-gradients of v should closely match the x-gradients of s.
3. Note that these could be solved while adding any constant value to v, so we will add one more objective: minimize $(v(0,0)-s(0,0))^2$. The top left corners of the two images should be the same color.

We use scipy.sparse for large sparse matrix computation. We use scipy.sparse.linalg.lsqr as the least-square solver.

Output:

Blending Results

Poisson Blending

Note: Row images are in the order: Target Image, Source Image, Naive Blending, Poisson Blending

Possible Reasons for bad results:

1. The pixel intensities of the source and the target image have different frequencies, such cases tend to affect the color of the source image while Poisson Blending. This is clear from the boat, boston source, target images.
2. Mask alignment: Aligning accurate masks like face keypoints is important step for efficient blending. In the Mona Lisa and Trump case above, unaligned face points produce poor results.

Mixed Blending