Assignment 5

Overview

In this assignment, a few different techniques that requires the manipulation of manifold of natural images are implemented. First, a pre-trained generator is inverted to find a latent variable that closely reconstructs the given real image. In the second part, a hand-drawn sketch is used to generate an image that fits the sketch accordingly. Finally, given an input image and a prompt images are generated using stable diffusion.

1: Inverting the generator

All natural images lie on a low-dimensional manifold. The output manifold of a trained generator can be considered close to this natual image manifold. Thus, we solve the following optimization problem \[ z^* = argmin_z \quad \mathcal{L}(G(z), x) \] Ablations on different loss functions are performed along with different choice of generative models like stylegan and vanilla GAN to get the best possible image reconstruction.

Experiment 1: Loss combinations

Weighted average of "Perceptual" and "L1" loss is tested and the results are shown below.

Increasing the weight of perceptual loss increases the artifacts as seen from the bottom right image. Also L1 is not sufficient to capture the details of the image and outputs are often blurry. Based on the above experiments L1 loss with weight 10 and Perceptual loss with weight 0.01 seems to work the best. The optimization is unstable and requires tuning of seed values to get the best results.

Experiment 2: Different generative models

Results for StyleGAN v/s vanilla GAN are compared below

StyleGAN outperforms Vanilla GAN. The reconstructed images from styleGAN are sharper and contain less artifacts

Experiment 3: Different latent spaces

Results for different latent spaces z, w and w+ are compared below

Best results are obtained from w+ latent space. The z latent space introduces some artifacts around the eyes and the w latent space is not able to capture the details of the image like whiskers.

On average the model takes about 25 secs per image to optimize.

2. Scribble to Image

Here we constrain the image generation process to fit a hand-drawn sketch. The constraints are assumed to be of the form \( f_i(x) = v_i \) for some scaled valued functions \( f_i \) and scalar values \( v_i \). With a trained generator, the optimization problem is \[ z^* = argmin_z \quad || M * G(z) - M * S ||_1 \]

Samples of sketches and their outputs

Using the common colors in the sketch leads to better results as opposed to using random colors. Also it can be seen that sparse sketches lead to more detailed outputs whereas dense sketches often lead to blobs and blurry images. Also with sparse sketches, the images tend be more staturated in color

3. Stable Diffusion

Following SDEdit, we use the input image as a "guide" by first transforming it into noise through forward diffusion and using it instead of a random sampling. This noised input image is then iteratively denoised using classifier-free diffusion guidance.