Overview

In this assignment, we are trying to applying gradient domain techniques to blend two images together in a more seamless way. Given a foreground image, a background image, and a binary mask, we would like to superimpose the foreground image onto the background image in a way that the boundary between the two images is not visible.

To achieve this, we want to ensure that at the boundary, the gradient between the foreground and background pixels matches the gradient of either the foreground or the background image. In addition, in the interior of the foreground image, we want to preserve the pixel gradient. This will yield an image that has a smoother transition along the boundary, but may vary in pixel values in the interior.

The objective function to optimize can be written as:
where \(d_i\) can be directly set to \(s_i - s_j\), or using a mixture of gradients from the source and target images: \(s_i\) and \(t_i\) are the pixel values of the source and target images, and \(N_i\) is the set of neighbors of pixel \(i\). \(S\) is the mask region. \(\boldsymbol{v}\) is the blended image in the mask region we want to optimize, where pixels outside the mask region are simply copied from the source image.

We can solve this least square problem by constructing a sparse coefficient matrix \(A\) and a right-hand side vector \(b\), and solve it using a sparse linear solver like scipy.sparse.linalg.lsqr. The values are solved for each color channel independently. One may also apply the gradient domain techniques to other problems by adding additional constraints to the objective function.

Toy Problem

The reconstructed image and the original image are shown below.

Poisson Blending Results

We follow the loss function described above to solve the Poisson blending problem. We only compute the horizontal and vertical gradients once and construct two masks for pixels that are inside the mask region and on the boundary. We can then form the coefficient matrix by filling in the coefficients for each pixel in these two masks, and the target vector by computing the corresponding gradients, adding or subtracting the background pixel if on the boundary.

The foreground image have relatively similar pixels around the mask boundary as the background image, hence the Poisson blending result seems reasonable, which is also much more natural than the naive approach. However, the difference in pixel values between the foreground and the background at the boundary causes the blended foreground to tint towards blue, given that the foreground boundary is whiter than the background, which has low gradient at the boundary.

To mitigate this, one may try to tune the color at the boundary of the foreground image or add additional loss terms to make the blended pixels similar to the original foreground pixels (but with a weight smaller than one). The additional loss does not help much in this case however and may break the solver sometimes. Using mixed gradient also leads to worse results as we want the blended region still follows the gradient of the original foreground image.

Additional Results

The Poisson blending results can show artifacts like unnatural color, given that we do not enforce any constraints on the color of the blended pixels. In this case, the loss term for the boundary may try to make the blended boundary pixels similar to the background pixels, which may lead to a color shift after this effect is spread to the interior pixels via the first term of the loss function. This artifact is more pronounced when the background is complex at the mask boundary and does not match well with the foreground image.

Bells & Whistles

Mixed Gradients

We added the mixed gradient by choosing the gradient from either the source or the target image, depending on which one has a larger magnitude. The results are shown below.


In the result shown, the foreground has zero gradient within and outside the logo, thus the Poisson blending result also stays flat, ending up with a blurred, unnatural look. The mixed gradient result, however, preserves the texture of the wall by following the gradient of the background image.

Color2Gray

We implemented a Color2Gray method by converting the RGB image to HSV and extract the saturation channel. The saturation image is used as the foreground, while the vanilla grayscale image is considered as the background, and the mask is the binarized saturation image given some threshold (0.1 in our example). We then apply the Poisson blending to the two images. This allows us to keep the grayscale intensity for regions with low saturation, while making the high-saturation pixels follow the mixed gradient of the saturation and the gray images. The loss term for the boundary and the mixed gradient serve to scale the new intensity closer to the original grayscale image for regions where saturation does not vary much.