Overview
This project centers around gradient-domain processing, specifically tackling the task of blending images together. The primary objective is to address the inherent issue of visible seams that arise when directly copying pixels from a source image to a target image, even under well-matched backgrounds. The core principle guiding our approach is based on the fact that human perception places greater emphasis on image gradients than overall intensity. Consequently, the problem is reformulated as an optimization task to determine target pixel values that maximally preserve the gradient of the source region while leaving the background pixels unaffected. It's noteworthy that this deliberate choice ignores overall intensity, allowing for potential color shifts in the blended region. The methodology revolves around precise manipulation of gradients to achieve a seamless transition between the source and target images. Poisson blending, by prioritizing gradient preservation, ensures a visually coherent output from disparate images.
Toy Problem
The toy problem is aimed at understanding the underlying equations used in gradient processing and setting up the optimization problem. Given the intensity of the source image at pixel \( (x, y) \) denoted as \( s(x,y)\) and the values of the output image to solve for as \( v(x,y) \), the objective is to formulate an optimization problem that minimizes the discrepancy between the gradients of the source image and the predicted values of the output image.
Minimize the squared differences between the x-gradients
\[
\text{Minimize} \quad E(v) = \sum_{x,y} \left( (v(x+1,y) - v(x,y)) - (s(x+1,y) - s(x,y)) \right)^2
\]
Minimize the squared differences between the y-gradients
\[
\text{Minimize} \quad E(v) = \sum_{x,y} \left( (v(x,y+1) - v(x,y)) - (s(x,y+1) - s(x,y)) \right)^2
\]
Equality constraint
\[
v(1,1) = s(1,1)
\]
Since the optimization can solve for \(v\) upto any constant we further constrain it by equating the top-left corners
of the two images. For the toy problem, this helps us recover the original image.
The above optimization problem can be solved as a linear least squares problem on \(v \). We can construct a sparse matrix
\(A \) of size \( (2\times H \times W + 1, H \times W)\), where the first \( H \times W \) rows correspond to the x-gradients
and the next \( H \times W \) rows correspond to the y-gradients and use the scipy.linalg.lsqr function to get \( v\).
The results can be seen below:
Poisson Blending
Poisson blending is an extension of the toy problem described earlier. In this case we compute gradients with respect to 4 neighbors of a pixels rather than just 2. Thus we are interested in solving the following optimization problem: \[ v = \text{argmin}_v \sum_{i \in S, j \in N_i \cap S} \left( (v_i - v_j) - (s_i - s_j) \right)^2 + \sum_{i \in S, j \in N_i \cap \neg S} \left( (v_i - t_j) - (s_i - s_j) \right)^2 \] The first part of the equation ensures consistency in gradients within \(S \), the latter part addresses scenarios where a pixel \(i\) is inside \(S\) while its neighboring pixel \(j\) is outside of \(S\). This is denoted by \( \neg S\). In such instances, \(v\) is not directly optimized for pixel \(j\) since it falls outside the defined region \(S\). To resolve this, we directly utilize the intensity value from the target image, denoted as \(t_j\) . Importantly, outside the region \(S\), we refrain from altering the target image \(t\); therefore, \(v_j\) in this context is not a variable but precisely corresponds to \(t_j\). Once again, the above equation reduces to a linear least squares problem and can be solved using an off-the-shelf solver.
Results1. Penguin on the road
2. Dolphin in the sky
3. Polar bear in the snow
Workarounds:
Infeasible Optimization: Naively solving the least squares problem is intractable for large images and will take a long time.
Sparse matrices are instead used to efficienctly compute the solution.
Out of range: There is no guarantee that the solution of the optimization problem will be within the intensity range of \( [0, 1] \).
Sometimes the values are observed to be negative as well as, greater than 1. A simple clip on the values seemed to do the trick without affecting the
visual quality of the image. I also tried rescaling and shifting the intensity values but it did not make much of a difference.
More results:
4. Gollum at the council of Elrond (Failure case)
The color and skin tone of gollum changed completely. This is a limitation of the poisson method as it ignores the raw color values of the original image and only cares about the gradients. Also, there was a sharp contrast between the foreground and background images which made it difficult for the poisson method to converge and blend the images seamlessly.
5. Face swaps
Bells and Whistles
Mixed Blending
Mixed blending is very similar to poisson blending. The only difference being that for each pixel it chooses a gradient either from the source image or the target image based on the magnitude of the gradient. This is useful in cases where source image has a plane texture lacking any gradients eg: text on solid background. In such cases, mixed blending will choose the gradient from the target image and give better results. The results can be seen below:
