Link Search Menu Expand Document
Learning Based Image Synthesis

Gradient Domain Fusion - Homework 2

Toy Problem

Here’s the source image:

Here’s what we will refer to as Source, target, and final image

Constructing the Sparse Matrices

We are trying to create a new image (v) from old image (s) whose pixel values are found by solving an optimization problem with 3 different kinds of constraints:

  • Make x_gradient of both images as similar as possible (( v(x+1,y)-v(x,y)) - (s(x+1,y)-s(x,y) ))^2 between (v) and (s)
  • Make y_gradient of both images as similar as possible (( v(x,y+1)-v(x,y)) - (s(x,y+1)-s(x,y)) )^2 between (v) and (s)
  • Make top left corners of both images same (v(1,1)-s(1,1))^2

In total, if we count the pixels and constraints:

  1. Each pixel (num_pixels_total = imh*imw) contributes one constraint for the x-gradient and one for the y-gradient
  2. The boundary pixels which only contribute one constraint each (either x-gradient or y-gradient, but not both)

Since we’re solving in the least squares sense, let’s construct an equation Ax = b (or Ax - b), where A is a matrix and x is the vector of solutions for this equation

Specifically:

  • A is a matrix whose:
    • num_rows = number of constraints (imh*(imw-1) + imw*(imh-1)) + 1
    • num_columns = number of variables (just the number of pixels in our case = imh*imw)
  • b is a vector of differences in pixel intensities for the x and y gradients of the source image, as well as the intensity of the top left pixel.
    • therefoer the shape of b = (same as num_rows in A) = (imh*(imw-1) + imw*(imh-1)) + 1

Finally we have:

A = lil_matrix((imh*imw + imh + imw, imh*imw))
b = np.zeros(imh*imw + imh + imw)

Book-keeping

We need a variable (let’s call it im2var) which serves as a mapping between each pixel in the image and a variable in the optimization problem.

In the context of this problem, each pixel in the image corresponds to a variable in the least squares problem. The im2var array is used to keep track of which variable corresponds to which pixel.

Implementation 

def toy_recon(im):
    """
    image: (H, W) 2D numpy array in range [0, 1]
    """
    imh, imw = im.shape
    im2var = np.arange(imh * imw).reshape((imh, imw))

    num_constraints = (imh*(imw-1) + (imh-1)*imw) + 1
    num_variables = imh * imw

    A = lil_matrix(arg1=(num_constraints, num_variables))
    b = np.zeros(num_constraints)

    e = 0
    # Objective 1 and 2
    for y in range(imh):
        for x in range(imw-1):
            A[e, im2var[y, x+1]] = 1
            A[e, im2var[y, x]] = -1
            b[e] = im[y, x+1] - im[y, x]
            e += 1
    for y in range(imh-1):
        for x in range(imw):
            A[e, im2var[y+1, x]] = 1
            A[e, im2var[y, x]] = -1
            b[e] = im[y+1, x] - im[y, x]
            e += 1

    # Objective 3
    A[e, im2var[0, 0]] = 1
    b[e] = image[0, 0]

    v = lsqr(A, b)[0]
    output = v[im2var].reshape((imh, imw))

    return output

Toy Problem Output

Poisson Blending

Here we operate on a slightly different objective function than the toy problem. The function is shown below:

Poisson Blending Results

Test Case

The images of the bear (sample) and swimming pool (target) were previously given. The outputs for the same are shown below:

Source Image Target Image

Final Blending

My Favourite Blend

  • I used a few pictures of my friends (with their permission) and got some fun outputs
  • The poisson blending works by minimizing the difference in gradients between source image and final blended image
Source Image Target Image

Final Blending

  • The blend is not perfect and there is still some difference in gradients which makes blend have some boundary artificats around the source

Additional Results

Working Case

Source Image Target Image

Failure Case

  • In this case, I tried overlaying the cat which has mostly white background and white texture onto a background with very different textures and colors
  • Due to above reasons, gradient correction changes the color of the sample too much making it infeasible

Bells & Whistles

Mixed Gradients

I tried mixed blending on one of the previous images to see if any visible improvements can be seen:

Plain Poisson Blending Blending with Mixed Gradients