Assignment 2

Image blending involves merging a segment from a "source" image into a "target" image. An initial approach to this task is straightforwardly pasting the segment onto the corresponding area. However, even with precise mask annotations, seamlessly pasting a segment is not a simple task.

A more effective solution is offered by gradient-based fusion. The concept is that pasting a segment onto another image creates noticeable gradient discrepancies, which our brain can detect. To address this, we locally adjust the segment to achieve smooth gradients along its borders (the surrounding region) and maintain consistency with the gradients within the segment.

This concept leads to Poisson blending, where the objective is to minimize the following equation:

$ \mathbf{v} = \text{arg} \min_v \sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - (s_i-s_j))^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - (s_i-s_j))^2 $

Here, $S$ represents the source region, $i$ denotes a point belonging to $S$, and $j$ is a neighbor of $i$ in terms of the x and y axes. Furthermore, $v$ denotes the pixel intensities of the merged segment, $s$ represents its original values, and $t$ represents the pixel values of the target.

Diving deeper into the equation, $v_i - v_j$ represents the local image gradients, which we aim to minimize the distance from the original source image gradients $s_i - s_j$. Along the border, we aim to preserve the intensity values of the target image, hence we compute the gradients considering the fixed $t_j$ values.

A variation of this approach is mixed gradient blending. While the methodology remains the same, we select the gradients from which to optimize the distance. Specifically, we compare the magnitudes of $s_i - s_j$ and $t_i - t_j$, and retain the maximum, denoted as $d_{ij}$. Consequently, the objective function becomes:

$\mathbf{v} = \text{arg} \min_v \sum_{i \in S, j \in N_i \cap S}((v_i - v_j) - d_{ij})^2 + \sum_{i \in S, j \in N_i \cap \neg S}((v_i - t_j) - d_{ij})^2 $

Mixed gradients are particularly beneficial when blending transparent objects, as demonstrated in our results. Finally, since color images have multiple channels, we solve an optimization problem for each channel separately. The implementation relies on SciPy to solve a least-squares problem.

Toy Problems

The range of values in the blended image goes outside [0, 1] so we try both rescaling and cliping the values - cliping seems to give more robust results. The bad results in the last result is mainly due to high gradient difference between the two images. Another problem we face is the change in color of the source images.