Assignment 5 - Cats Photo Editing

• Sihan Liu

0. Project Overview

In this assignment, I need to use various image manipulation techniques on natural images. The first part will involve implementing the inversion of a GAN generator and finding a latent variable that can accurately reconstruct a given real image. Then, I need to generate an image that fits a hand-drawn sketch. For the third part, I need to implemented img2img with pretrained stable diffusion. For Bells & Whistles, I interpolated between two latent codes in the GAN model, and generate an image sequence. Also, I used the pretrained stable video diffusion to add motion to static cat images.

1. Inverting the Generator

In this part, I try to reconstruct the image from a particular latent code. To implement this task:

python main.py --model vanilla --mode project --latent z
python main.py --model stylegan --mode project --latent w+

Visualization of the results of different loss combinations (default: stylegan, w+):

Epoch	250	500	750	1000
Preceptual Loss
Preceptual Loss (0.01) + L1 Loss (10)
Preceptual Loss (0.01) + L2 Loss (10)

Visualization of the results of different generative models and different latent space (default: Preceptual + l1). I show different models with the same target and also more examples with stylegan w+:

Epoch	250	500	750	1000
Vanilla (z) (t = 11.66 s)
StyleGAN (z)
StyleGAN (w) (t = 26.15 s)
StyleGAN (w) (more examples)
StyleGAN (w+) (t = 26.48 s)
StyleGAN (w+) (more examples)

Comments:

• The perceptual loss is a good choice to generate a reasonable and realistic image, while l1/l2 loss can keep the original structure of cats. For best result, I adopt the perceptual loss with l1 loss. Uing w and w+ achieve similar time spent. But using StyleGAN and latent space w+ achieves the best performance becuase it outcomes more detailed and preserve good color images. One possible reason is that w+ integrates different w vectors.

2. Scribble to Image

From the results we can easily see that the generated images are not very realistic from the corresponding scribbles. The main reason is that most scribbles are too simple and the model cannot learn the mapping from the scribbles to the real images. For some denser scribbles, the model still cannot generate a reasonable image. The main reason is that the model is not general after training on a limited dataset. If we train the model on a larger dataset, the model may be able to generate better images. Though not realistic, the results still follow the basic colors from the scribbles. The results are better when the scribbles are denser. For example, the results of scribble 5 and 4 are better than the results of scribble 1 and 2.

To implement style transfer:

python main.py --model stylegan --mode draw --latent w+

Epoch	Origin	250	500	750	1000
Scribble 1
Scribble 2
Scribble 3
Scribble 4
Scribble 5
Scribble 6
Scribble 7
Scribble 8
Scribble 9

Part 3: Stable Diffusion

We do img2img with pretrained stable diffusion.Below are the results of generations with different strengths and number of timesteps. We can see that the stronger the strength, the more detailed the generated images are, and the more noise there is, the more the generated images match and fit the input.

Input	Timesteps=500, Strength=15	Timesteps=1000, Strength=15	Timesteps=500, Strength=10
"Grumpy cat reimagined as a royal painting"
"a colorful cat as a oil painting"

Bells & Whistles

Interpolate between two latent codes in the GAN model, and generate an image sequence (2pt)

The start and the end of each interpolation are two different cats which are origin 1 and origin 2 respectively, and the interpolation process is smooth and reasonable. The interpolation gradually changes the cat's color, shape, and background, which is a good demonstration of the latent space of StyleGAN. Visualization of the results:

	origin 1	origin 2	Interpolation
Example 1
Example 2

Other brilliant ideas you come up with. (up to 5pts)

I use pretrained stable video diffusion to add some motion to the static input cat image.

	input image	output
Example 1
Example 2