Learning Based Image Synthesis (16-726)

Assignment 2

Aviral Agrawal (avirala)

1. Overview

This project delves into the realm of gradient-domain processing that has been employed in applications such as image editing, computer graphics, and computer vision.

From the Figures 1, 2, 3 we see that simply cropping out a region of interest and pasting it into a background isn't quite appealing and would produce rather pronounced seams. To deal with this issue, we manipulate the gradients of pixels within an image rather than directly manipulating the pixel values themselves. This approach allows for more nuanced and visually appealing results.

Blending images can be achieved by a gradient-domain processing algorithm known as "Poisson blending." This technique involves seamlessly merging an object or texture from one image into another by considering the gradients of both images and minimizing the differences between them.

• First we see, with the help of a toy example, how we can reconstruct an image from its gradients.
• We then implement Poisson blending to seamlessly merge an object or texture from a source image to a target image.
• However, the Poisson blending algorithm is not without its limitations and we shall use mixed gradients to overcome some of these limitations.
• Finally, we shall explore the use of mixed gradients to efficiently convert color images to grayscale without losing contrast information.

2. Toy Problem

We see, with the help of a toy example figure below, how we can reconstruct an image from its gradients. We have selected the first pixel value as the boundary condition in order to reconstruct the image.

3. Poisson Blending

We implement Poisson blending to seamlessly merge an object or texture from a source image to a target image. The Poisson blending algorithm involves solving a linear system of equations to minimize the differences between the gradients of the source and target images. The algorithm can be summarized as follows:

• First, identify the region of interest, 'S' from the source image.
• Next, we create a binary mask to indicate the region of interest in the source image.
• We then identify the region in the target image, 'v', where we want to blend in the source region
• We then solve a linear system of equations to minimize the differences between the gradients of the source and target images, while respecting the boundary conditions defined by the mask.
• Finally, we reconstruct the blended image from the gradients using the solved linear system.

The linear system of equations can be formulated as follows: $$ v = \arg\min_v \sum_{i \in S, j \in N_i \cap S} \left( (v_i - v_j) - (s_i - s_j) \right)^2 + \sum_{i \in S, j \in N_i \cap \neg S} \left( (v_i - t_j) - (s_i - s_j) \right)^2 $$
While selecting the images for Poisson blending, we try to select images that have similar backgrounds since that results in similar gradients and hence, a more visually appealing Poisson blending result.
We shall now see the results of Poisson blending on a few images. FAILURE CASE We see that the Poisson blending algorithm is not without its limitations. The algorithm can produce visually unappealing results when the source and target images have significant differences in their gradients. In the example below, the source image (halloween pumpkin) has a very different gradient compared to the target image (Oberoi hotel). Thus, when solving for the Poisson blending linear system, the boundary conditions and the gradient constrainsts lead to a changed color of the blended region.

4. Bells & Whistles : Mixed Gradient Blending

A minor change to the Poisson blending algorithm is called the mixed gradients approach, wherein we blend the source and target image by considering the maximum of the source of target gradients. In the following example we can see that Poisson blending produces a blur boundary around the blended region while with the use of mixed gradients we can overcome this limitation.

5. Bells & Whistles : Color2Gray

We shall now explore the use of mixed gradients to efficiently convert color images to grayscale without losing contrast information.

• First, we convert the color image to the HSV color space.
• Next, we convert the color image to grayscale using the mixed gradients approach with the saturation channel as the source region and the value region as the target region.
• The result of the linear solver is the output grayscale image.