**Gradient Domain Fusion** Introduction =============================================================================== Background ------------------------------------------------------------------------------- The objective of this assignment is to seamlessly blend one image into another. Naively blending two images results in visible "seams" due to mismatch in the brightness and colour profile of the two images - making the end-result look uncanny. Gradient domain fusion is one method to address this. The intuition behind this is that while blending two images, we don't really care about the overall brightness/intensity of the source image, but the image gradients. We are interested in how the intensity changes through the source image. Once we have the intensity gradients and the colours at the seams/boundaries from the target image, we can "integrate" the gradients to get a blended result that has the colour characteristics of the target image extrapolated into the source image. Here, we will explore a popular gradient domain fusion method - poisson blending. Overview ------------------------------------------------------------------------------- Our main task is to implement Poisson blending, and then show another application of gradient domain fusion in colour to grayscale conversion. But before we do that, here is what an image looks like we blend it naively by just copying pixels from the source image to the target image.

We want the image of the bear (source) to merge seamlessly with the pool (target). So instead of the naive method, we will use the gradients from the image of the bear to "extrapolate" the colour of the pool into the image of the swimming bear. So, next we will discuss poisson blending to explain how we will achieve this. Method =============================================================================== Poisson Blending ------------------------------------------------------------------------------- Poisson Blending can be formulated in the following way: Given pixel values of the source image $s$ and of the target image $t$, we want to solve for new intensity values v within the source region $S$. This can be depicted as a least square optimization problem: $v = argmin_{v} \sum_{i \in S, j \in N_i \cap S} ((v_i - v_j) - (s_i - s_j))^2 + \sum_{i \in S, j \in N_i \cap \neg S} ((v_i - t_j) - (s_i - s_j))^2$ where $i$ is a pixel in the source image, and each $j$ is a 4-neighbor of $i$. Each summation guides the gradient values to match those of the source region. In simple terms, the first summation forces the gradients inside the merged region to match the gradients of the source image and the second summation forces the pixel values at the boundary of the merged region to match the target image. The second summation can be interpreted as a boundary constraint, as $t_j$ are known pixel values from the target image at the boundaries. Now this method obviously has limitations and some caveats. To get a visually appealing result, we would want the source image and the target region to have similar colours and textures. For example this method will work well in the example shown above as the bear is surrounded by water already (albeit of different colour). Implementation details ------------------------------------------------------------------------------- We formulate the least square problem in the standard matrix form $(Av -b)^2$. We create the rows of $A$ one by one using scipy's `sparse csr_matrix`. We use this formulation as each pixel will add its own row in $A$ and only two entries in each row will be non-zero. Without the sparse formulation, $A$ will be huge and take a lot of memory, making the least square solution very slow. The equation will be overdetermined, as we will add also add the boundary constraints, and each boundary constraint will add a row in $A$. $b$ is similarly filled with gradient values of the source pixels, and later on, pixel values from the target image for the boundary conditions. We use scipy's `lsqr` solver to solve this overdetermined system with `atol` and `btol` set to $1e^{-10}$. This system has to be solved independently for each colour channel. Another important implementation detail is that since the merging mask can be of any arbitrary shape, we need to take care of edge cases and use a dictionary to map the pixel coordinates to an index in $v$. Once the overdetermined system is solved, the solution $v$ will have exactly the same number of elements as the number of pixels in the merge region. We will use the dictionary to pick values from vector $v$ and place them in the target merge region. Results =============================================================================== Toy Problem ------------------------------------------------------------------------------- We first try out our poisson solver on a small image. We try to reconstruct the image using its gradients and only one pixel value as the "boundary condition". We arbitrarily use the $[0,0]$ pixel as the known value. We don't have a separate source and target image in this toy example.

Poisson Blending ------------------------------------------------------------------------------- We will first see the result of poisson blending on the example image. We see that the bear has perfectly blended into the pool.

Next we will see a series of results on our own images. In each case, we will first show the source and target image, and then the results.

---

---

---

---

The one shown below is a failure case. The merge failed due to the drastic mismatch in colours and brightness. The inteention was to blend the logo with the building, so that it kind of appears to float in front of it. This issue is solved once we used mixed gradients in the Bells and Whistles section.

---

Bells and whistles =============================================================================== Mixed Gradient ------------------------------------------------------------------------------- Here we use gradients from the source image or the target image, depending on which one has a larger absolute value at that pixel. This can sometimes lead to an interesting transparent overlay effect.

Color2Gray ------------------------------------------------------------------------------- Certain colour images are hard to convert to grayscale, like the images in the colour-blindness test. Off-the-shelf methods from opencv and skimage fail due to the low contrast difference between the colours. Here our gradient based method can be advantageous. If we convert the RGB image to HSV space and then use the gradients from the Saturation channel to re-construct the image, we can get a better gray-scale image. To keep the intensities similar to a normal grayscale image, we initialise one pixel from the normal grayscale image while solving the least square problem. Here we refer to the opencv grayscale converted image "normal".