16726-Learning Based Image Synthesis-Spring 24

Assignment #2 - Gradient Domain Fusion

Student Name: Peipei Zhong, Andrew ID: peipeiz

Brief Description

The assignment focuses on gradient-domain processing, specifically "Poisson blending," a technique for seamlessly blending objects or textures from a source image into a target image. Unlike simple copy-paste methods, which often result in noticeable seams, Poisson blending aims to preserve the gradient of the source region within the target image, disregarding overall intensity. This approach is formulated as a least squares problem, where the objective is to match the gradients in the blended area to those of the source, ensuring a natural integration without altering the background pixels of the target image.

Poisson Blending Method

Poisson Blending uses the Poisson equation to achieve seamless image blending, with the basic idea of smoothly integrating the source image region into the target image while preserving the local gradient information of the source image. The key to Poisson Blending lies in solving the following optimization problems:

v = \arg\min_v \sum_{i \in S, j \in N_i \cap S} ((v_i - v_j) - (s_i - s_j))^2 + \sum_{i \in S, j \in N_i \cap \neg S} ((v_i - t_j) - (s_i - s_j))^2

$v_1$ $v_j$ $s_1$ $s_j$ are the pixel values of the source image at positions i and j.

$v_1-v_j$ $s_1-s_j$ . This ensures that the features (such as edges and textures) in the source image are preserved in the mixed image.

$t_j$ is the pixel value of the target image at position j.

The purpose of this expression is to smoothly transition at the edges of the source image region, so that the boundaries of the mixed region naturally blend with the target image, reducing visual discontinuity or mutation.

Toy Problem

The purpose of toy problem is to restore an image.

We have three objectives:

$\left((v(x+1,y) - v(x,y)) - (s(x+1,y) - s(x,y))\right)^2$
$\left((v(x,y+1) - v(x,y)) - (s(x,y+1) - s(x,y))\right)^2$
$\left((v(1,1) - s(1,1))\right)^2$

Result:

Blending Result

Success

Example

the source and target image

the blended image with the source pixels directly copied into the target region and the final blend result

My favorite

the source and target image

the blended image with the source pixels directly copied into the target region and the final blend result

Another successful example

the source and target image

the blended image with the source pixels directly copied into the target region and the final blend result

Failure

the source and target image

the blended image with the source pixels directly copied into the target region and the final blend result

I guess that when the gradient of the source image and the gradient of the target image around the mask are very different, the resulting blend effect is not good. Because at this time, it is necessary to maintain the internal gradient of the source after the change, and also make the edge gradient of the mask consistent with the gradient of the target.

Extra credit-Mixed Gradient

There are some limitations in naive poisson blending, and we can solve them by using the gradient in source or target with the larger magnitude as the guide, rather than the source gradient:

v = \arg\min_v \sum_{i \in S, j \in N_i \cap S} ((v_i - v_j) - d_{ij} + \sum_{i \in S, j \in N_i \cap \neg S} ((v_i - t_j) - d_{ij}


if abs(s_i - s_j) >= abs (t_i - t_j):
    d_ij = s_i - s_j
else:
    d_ij = t_i - t_j

the source and target image

Mixed Gradient Method: the blended image with the source pixels directly copied into the target region and the final blend result

The result is good.

Naive Blend Method: the blended image with the source pixels directly copied into the target region and the final blend result

The result is not so good, the space between the 'welcome' letters is obviously blurred and does not blend well with the background.