Brief introduction
This project explores poisson blending, a technique to combine images together, and image gradient domain adjustment.
Look at the following equation, which is the objective function of poisson blending. \[ v = \arg\min_v \left\{ \sum_{{i \in S, j \in N_i \cap S}} ((v_i - v_j) - (s_i - s_j))^2 + \sum_{{i \in S, j \in N_i \cap \neg S}} ((v_i - t_j) - (s_i - s_j))^2 \right\} \] Here, \(v\) is the result image(combined result image), \(s\) is the source image(the full-resolution image of our inserted segement), \(N_i\) is the set of neighbors of pixel \(i\), \(S\) is the region where we are editing, and \(t\) is the target image, i.e. the background.
We could divide the equation of two parts by the two sums. The first part restricts that our segement should be consistent insides \(S\). While the second part cares about the edges, which makes the gradient domain smooth and continuous.
Toy problem
In the toy problem, what we really do is to reconstruct an image. Denote the intensity of the source image at (x, y) as s(x,y) and the values of the image to solve for as v(x,y). The objective is to minimzie \((( v(x+1,y)-v(x,y)) - (s(x+1,y)-s(x,y)) )^2\) and \((( v(x,y+1)-v(x,y)) - (s(x,y+1)-s(x,y)) )^2\) , so the gradient domain matches horizontally and vertically.
As the instructions go, this is a least square problem, and we could solve it by solving a linear system. We could write the equation as \(Av = b\), where \(A\) is a sparse matrix, and \(b\) is a vector. For each pixel, we add two constraints into matrix A that restricts its gradient domain.
Note that in order to make sure the amount of constraints, I used max and min to make sure that the neighbors are in the range of the image. Finally add the constraint that the foremost pixel should be the same as the source image.
Poisson blending
When it comes to poisson blending, the problem actually becomes much easier. According to the equation above, we see that the first summation is pretty similar with what we did in the toy problem. The only difference is that, in the second sum, we have a third variant \(t\). Since the unknown variable is \(v\), we could treat \(t\) as a constant, and put \(t\) right side with \(b\) in the linear system. Here is the result.
My favorite results
Failure cases
As depicted in the image, the result is not very good. I guess it's because poisson blending behaves poor when the background and foreground both have a rather complex texture. Like the concert hall, the color and texture of the wall, seats, and the stage are not consistent, which makes the result not ideal.
I also tried to blend images with different color styles. The result again shows that poisson blending is sensitive to background of objects, and the surrounding of the dog's head is darkened.
Bells & Whistles
Mixed Gradient
For naive poisson blending, when we are tackling with area \(S\), we entirely depend on the gradients of the source image, ignoring that of the background. This might cause the result to be not very smooth. Therefore, we take the bigger gradient of the source and the target, and use it as the reference.
Here are a pair for comparison. After applying mixed blending, the lines of the water appears more natural.
Color2Gray
Since existing color to gray functions sometimes lose the contrast of the image, making it hard to recognize such as the color blindness test images. I implemented a color2gray function with mixed poisson blending. I first explored hue, saturation, value channels of the image.
Among them, s channel best captures the saturation of the image, and v channel represents the overall image well. Therefore, I input s and v channels into my mixed blending function, and the result is as follows.
