Project Summary

When trying to combine two images (a background image and a source object), simply overlaying the source onto the background is not enough. This gives a very unrealistic look, where the brightness is most likely wrong as well as the sudden change in the borders making the image very evidently fake. To remedy this, we can use a method called Poisson Blending. The main idea behind Poisson Blending is that when overlaying two images, the main feature we need to fix is the change in gradients, not neccesarily the pixel intensity of the images. So, we can setup a least squares systems of equations problem where we try recreating our source object in the appropriate region in the background image, but instead of just overlaying the two, we try to make the gradients of the source object and the background image match as closely as possible. This ends up working fairly well in many cases, and gives us the results shown below.

To implement Poisson Blending, I constructed a sparse matrix (SK's lil_matrix worked well) representing the equations presented in the writeup. I then used SK's lsqr function to solve the sparse matrix, running it once per color channel then stacking the results together. This resulted in an algorithm that runs in less than 10 seconds on average, on my Macbook Pro with an M1 Pro chip.

Toy Problem

The toy problem is essentially can we succesfully use this least squares formulation to reconstruct an image given just its gradients? Below I show the result of my algorithm, which faithfully reconstructs this image given just the gradients.

Example Results

Favorite Result

In the spirit of the F1 season starting soon, I decided to tailor generate some F1 themed results! This image is my personal favorite, since it seems like the algorithm does a good job of handling not only the blending but also the intensity change due to weird lighting. I didn't do anything special to make this happen. As a bonus, here is a text conversation of me (briefly) fooling my friend with the image! This ended up fooling quite a few more of my friends, showing the algorithm works well.

Result using mixed blending

My friend falling for my generated image

Other Results

Here is my favorite result, but using Poisson blending. Because my mask wasn't very tight, you can see that it blurs the borders which is ok, but not super realistic, which is where the mixed blending algorithm shines.

This was run using standard Poisson Blending. Compared to the naive implementation, the blending is quite good. It dims the very bright borders of the target and makes it fit in way more. However, there is still some noticeable artifacts around the feet.

This is the same result as above, but with mixed blending. Honestly I couldn't really tell the difference between the two results, probably because the mixed solution is not doing anything different from the poission solution in this case.

Failure Case

This is an image of a dimly lit person I found online. I am trying to overlay this on top of an art gallery piece using Poisson Blending. The idea is that it seems like Poisson blending doesn't work well when trying to overlay a dimly lit subject on a bright image, which is shown here. I think the main point here is that in certain scenarios it, not preserving the intensity of the original image is a drawback. You can see that some of the border "bleeds" into the person, and the change in intensity doesn't really add anything to this in terms of making it look more realistic.

This is the same test as above but using mixed blending. In an attempt to use these mixed gradients, it inadvertently blends part of the original art piece into my overlayed image, which is not ideal. However, it seems like the borders are slightly better in this example.

Bells and Whistles

Mixed Gradients

As shown in my above results, I implemented mixed gradients. This was pretty straight forward and was highly similar to my Poission blending solution.

Color2Gray

Here are some results of my Color2Gray method. I found that depending on the image, sometimes the "S" value was better to use, and sometime the "V" value gave better results The implementation involved converting the given RGB image to HSV, then passing in the respective value (either H, S, or V) along with the naive graysacle image as the background to my mixed_gradients function. Below are some example results:

Before

Result of picking the Saturation value and using mixed gradient blending.

Before

Result of picking the "Value" value and using mixed gradient blending.