Learning-Based Image Synthesis

Assignment 5: Cats Photo Editing

Andrew Id: shubhikg

Task Description

In this assignment, we explore different techniques that require us to manipulate images on the manifold of natural images. First, we inverted a pre-trained generator to find a latent variable that closely reconstructs the given real image. GAN inversion refers to the process of reconstructing an input image from its corresponding representation in the latent space of a Generative Adversarial Network (GAN). We then took a hand-drawn sketch and generated an image that fitted the sketch accordingly. In the last part of the assignment, we used a Stable diffusion model to generate images based on an input image and a prompt.

Part 1: Inverting the Generator

Inverting a generator in GANs involves finding the latent vector that, when fed into the generator network, produces an output similar to a given real image. This task is challenging due to the complex and nonlinear mapping between the latent space and the image space learned by the generator. Techniques for inversion include optimization-based methods that minimize the difference between the generated image and the target image, as well as encoder-decoder networks trained to map images to their corresponding latent codes. Successful inversion enables applications such as image editing, style transfer, and understanding the latent space's structure learned by the generator, contributing to advancements in image generation and manipulation.

1.1 Ablation on weight of losses

The objective of this experiment is to ablate the weight of different losses used in GAN inversion optimization. I used L1 loss, L2 loss and content loss using conv 5 layer of vgg and tried different weight combinations.

The results using stylegan gan with w+ latent are as follows:

Target Image	Perc = 0.0, L1=10, L2=0	Perc = 0.1, L1=10, L2=0	Perc = 0.01, L1=10, L2=0	Perc = 0.01, L1=10, L2=1	Perc = 0.001, L1=10, L2=0	Perc = 0.001, L1=10, L2=1

From the above results we can see that weights Perc=0.01, L1=10, L2=0 results are the best, while rest are detroiting in quality. Although the results are very similar for image 2, image 1 has very different results for different setttings. Hence using that comparison, for future experiments I plan to use these values.

1.2 Ablation on weight of GAN model

In this experiment, I experiemnted with different generator models. The results are:

Target Image	Vanilla GAN	Style GAN

From the above results we can see that Style GAN produces better output.

1.3 Ablation on latent space

In this experiment, I experiemnted with different latent space of generator models. The results are:

Target Image	Vanilla GAN: latent z	Style GAN: latent w	Style GAN: latent w+

From the above results we can see that Style GAN with latent space w+ produces the best output. The reason for this is that in w+ latnt space, there is seperate style vector for each layer of Style GAN.

Part 2: Scribble to Image

In this part of assignment, we implement generating images using color scribble as inputs. Color scribbling using GAN constraints involves guiding a Generative Adversarial Network (GAN) to generate an image based on user-provided color scribbles while adhering to the constraints imposed by the GAN's training. The user provides rough color annotations or scribbles on a canvas, indicating desired color regions or features. The GAN then incorporates these scribbles as additional input or guidance during the image generation process, ensuring that the generated image reflects the desired colors and features while maintaining realism and coherence learned during training.

The results are:

Input Image	Mask Image	Generated Output

From the above results we can see that the sparse input gives poor results while dense input gives better output. The model can better better quality results using dense input and can generate more realistic results. On the other hand sparse outputs are very blurry and non-realistic.

Part 3: Stable Diffusion

In this experiment, I generate images based on an input image and a prompt using stable diffusion. Originally Stable Diffusion synthesizes images from textual prompts alone. We modify the approach so that image input serves as a “guide,” transforming the given input image into noise through the forward diffusion process instead of starting with random sampling and then iteratively denoising to generate a realistic image using a pre-trained diffusion model. The results are:

The results are:

Input Image	Prompt	Output Image
	Grumpy cat reimagined as a royal painting
	House in the middle with sun and green grass

3.1 Ablation on strength

In this part of the assignment, I varied the value of strength for conditioning on diffusion model. The results on the grumpy cat image with the same prompt as aboveand 800 steps are:

Strength 5	Strength 10	Strength 20	Strength 40	Strength 60

From the above results we can see that as the strength increase, the output aligns less to the input image and more to the text prompt.

3.1 Ablation on number of steps

In this part of the assignment, I varied the value of the number of steps for diffusion model. The results on the grumpy cat image with the same prompt and 40 strength as above are:

300 steps	500 steps	800 steps	900 steps	1000 steps

From the above results we can see that as the number of steps increase, the output aligns less to the input image and more to the text prompt. This happens because as the number of steps increase, the input noisy vector carries less information from the input image and move more towards gaussian noise.