16-726 Learning-Based Image Synthesis

Spring 2024

Assignment #2 - Gradient Domain Fusion

Yifei Liu

I. Introduction

For this assignment, we explored gradient-domain processing and focus on Poisson blending. The goal is to seamlessly blend an object or texture from a source image into a target image by solving a least-squares problem, instead of simply copy-pasting the source to target image. The technique achieves this by ensuring the gradient of the blended region matches that of source image, while contrainted by the boundary conditions of target image.

II. Results

Toy Problem

I first implemented the Poisson blending on a toy problem: reconstructing the image v by computing the x and y gradients from an image s and using all the gradients plus one pixel intensity. The objective is:

The result is shown below.

Poisson Blending

For the actual Poisson Blending, we formulate our objective as a least squares problem (Av = b) and solve this blending constraints:

where S is the pixel intensities of the source image, T is the pixel intensities of the target image, and v is the new intesity values within the source region S.

Matrix A here is constructed as a sparse matrix and only pixels within the mask is set. I solved the least-squares problem for each of the 3 channels and stacked the result to the final_blend. I used the given helper code masking_code.py to generate the mask and run the implementation to blend the source to target images. The results are shown below.

Favorite blending result:

Source
Target
Wildfire Naive vs Final Blend

Additional results:

Source
Target
Sample Naive vs Final Blend
Source
Target
Cat Swim Naive vs Final Blend

Failure Example

Blending my cat to the room picture I found online does not work very well because although Poisson blending take into consideration gradients and consider the visual characteristics such as edges and textures of the blended region, it might not be able to directly align the tiles of the floor in my picture and the online picture without further processing, for example rotation and scaling.

Source
Target
My Cat Naive vs Final Blend

III. Bells & Whistles

Mixed Gradients

For mixed gradients, instead of directly using the source image's gradients, we compare the magnitude of gradients between the source and target images at each pixel and select the larger of the two. This method enhances the final blended image by preserving the most prominent features from both images. As shown in result, the writing on plain background is naturally blending into the target sky, while preserving the texture of the sky.

Source
Target
Quote in Sky Naive vs Final Blend

2. Color2Gray

I also implemented the enhanced color to grayscale function mixed_grad_color2gray() that involves blending the gradients of the Saturation (S) channel from the HSV color space with the gradients of a standard grayscale RGB image. By comparing the gradients in both S channel and grayscale, I select the larger of the two at each pixel to construct new gradient field. This mixed gradient field is then used to solve a Poisson equation, reconstructing a grayscale image that preserves the original image's contrast.