Assignment 5: Cats Photo Editing

Student name: Tianxiang Lin
Andrew ID: tianxian

Introduction

This project aims at:

(1) Inverting a pre-trained generator to find a latent variable that closely reconstructs the given real image.

(2) Taking a hand-drawn sketch and generate an image that fits the sketch accordingly.

(3) Generate images based on an input image and a prompt using stable diffusion.

Inverting the Generator

Show some example outputs of your image reconstruction efforts using (1) various combinations of the losses including Lp loss, Preceptual loss and/or regularization loss that penalizes L2 norm of delta.

The following results generates from StyleGAN on w+ latent space with 1000 iterations. From the following results, the baseline performs the best when perc_wgt=10, reg_wgt=100, l1_wgt=0, l2_wgt=0.

Name	perc_wgt	reg_wgt	l1_wgt	l2_wgt	loss
Original	N/A	N/A	N/A	N/A	N/A
StyleGAN	10	0	0	0	89.188019
StyleGAN	0	10	0	0	0.000000
StyleGAN	0	0	10	0	1.337266
StyleGAN	0	0	0	10	0.314818
StyleGAN	10	100	0	0	84.629807
StyleGAN	10	10	0	0	89.957237
StyleGAN	10	1	0	0	75.124519
StyleGAN	10	0.1	0	0	86.234268
StyleGAN	10	1	100	0	97.208191
StyleGAN	10	1	10	0	97.794884
StyleGAN	10	1	1	0	93.676056
StyleGAN	10	1	0.1	0	91.093567

(2) different generative models including vanilla GAN, StyleGAN.

The following results generates from vanilla GAN and StyleGAN on z latent space with 1000 iterations. The weights of losses are: perc_wgt=10, reg_wgt=100, l1_wgt=0.1, l2_wgt=0. From the following results, StyleGAN performs better than Vanilla GAN.

Name	perc_wgt	reg_wgt	l1_wgt	l2_wgt	loss
Original	N/A	N/A	N/A	N/A	N/A
Vanilla GAN	10	100	0.1	0	103.774223
StyleGAN	10	100	0.1	0	80.858635

(3) different latent space (latent code in z space, w space, and w+ space).

The following results generates from StyleGAN on z. w, w+ latent space with 1000 iterations. The weights of losses are: perc_wgt=10, reg_wgt=100, l1_wgt=0.1, l2_wgt=0. From the following results, StyleGAN performs better on w+ latent space.

Name	perc_wgt	reg_wgt	l1_wgt	l2_wgt	loss
Original	N/A	N/A	N/A	N/A	N/A
StyleGAN (z)	10	1	0	0	128.940598
StyleGAN (w)	10	1	0	0	92.629356
StyleGAN (w+)	10	1	0	0	75.124519

(4) Give comments on why the various outputs look how they do. Which combination gives you the best result and how fast your method performs.

From the experiments, the self regularization loss helps the prediction images be close to its original input. Also, from the empirical observations, l1 loss performs better than l2 loss because the predictions are sharp and clear. But here we disabled both l1 and l2 losses.

From the results above, StyleGAN is better than vanilla GAN and performs the best on w+ latent space with perc_wgt=10, reg_wgt=100, l1_wgt=0, l2_wgt=0. The performance time of all combinations is basically the same, which is within 20 seconds.

Scribble to Image

Draw some cats and see what your model can come up with! Experiment with sparser and denser sketches and the use of color. Show us a handful of example outputs along with your commentary on what seems to have happened and why.

The following results generates from StyleGAN on w+ latent space with 1000 iterations. The weights of losses are: perc_wgt=10, reg_wgt=100, l1_wgt=0, l2_wgt=0. From the results, we can see that the model predicted fair cat images on sparse sketches with sharp characters of cats, especially their eyes, while did a bad job on dense sketches. This may result from the regularization loss I set. The model will generate results not only structurally similar but pixelwise more constrained, giving the model less flexibility. Also, sparser sketches provides more details of cats, including eyes and mouths, giving model more information for predictions, which denser sketches cannot provide.

Sketch	Mask	Output

Stable Diffusion

(1) Show some example outputs of your guided image synthesis on at least 2 different input images.

Results from two input images with two different prompts are shown below.

Prompt	Timesteps	Noise	Strength
"Grumpy cat reimagined as a royal painting"	1000	N(0,1)	15.0
"Grumpy cat reimagined as an oil painting with blue background"	1000	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	700	N(0,1)	15.0
"Grumpy cat reimagined as an oil painting"	700	N(0,1)	15.0

(2) Furthermore, please show a comparison of generated images using 2 different amounts of noises added to the input

Results from two input images are shown below. To add different amounts of noises, the number of timesteps are set to 500 and 1000 for the input 1, 500 and 700 for the input 2.

Prompt	Timesteps	Noise	Strength
"Grumpy cat reimagined as a royal painting"	500	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	1000	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	500	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	700	N(0,1)	15.0

(3) 2 different classifier-free guidance strength values.

Results from two input images are shown below. To compare the results, I set the classifier-free guidance strength values to 10.0, 15.0, and 20.0 for 2 inputs.

Prompt	Timesteps	Noise	Strength
"Grumpy cat reimagined as a royal painting"	1000	N(0,1)	10.0
"Grumpy cat reimagined as a royal painting"	1000	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	1000	N(0,1)	20.0
"Grumpy cat reimagined as a royal painting"	700	N(0,1)	10.0
"Grumpy cat reimagined as a royal painting"	700	N(0,1)	15.0
"Grumpy cat reimagined as a royal painting"	700	N(0,1)	20.0

Bells & Whistles

Interpolate between two latent codes in the GAN model, and generate an image sequence (2pt)

For interpolation, 3 demos are shown in the following sheet. They are obtained from w+ latent space. (Please open the image in the new tab if the animation is static.)

Source	Target	Interpolation

Develop a cool user interface and record a UI demo (4 pts). Write a cool front end for your optimization backend.

Here I use tkinter to implement frontend turbo. The following screenshot shows how the frontend works.

Experiment with high-res models of Grumpy Cat (2 pts)

Here, I use StyleGAN256 for the task of interpolation. 3 interpolation demos are shown in the following sheet. They are obtained from w+ latent space. (Please open the image in the new tab if the animation is static.)

Source	Target	Interpolation