16-726 Assignment 5

David Krajewski - dkrajews

Project Summary

For this project, we first invert different GAN models to try and find latent codes that match given images as closely as possible. We then use this to try and provide the model with a sketch we want it to follow closely, allowing us to draw sketches and get somewhat realistic results back. Finally, we modify the Stable Diffusion sampling process to try and match a sketch as closely as possible, while being able to condition using text prompts.

GAN

Inversion

Losses

The following grid shows the result of GAN inversion using StyleGAN and the z latent space with different weights associated to the perceptual loss. Judging by the results below, it seems that a perceptual loss weight of 0.01 achieves a good balance between looking faithful to the original image while retaining most of the detail. It seems like weighing the content loss too heavily (i.e. weight of 1) results in the general structure seeming ok, but the details themselves are blurry / not accurate, likely due to the L1 pixel space loss being weighed less. On the other extreme, weighing the content loss not enough results in some of the finer details looking good (e.g. eyes / mouth, general color, etc. ), while the structure / pose being worse / more blurry. Again, this is likely due to the pixel space loss being weighed too heavily and the content loss being not as emphasized.

0.0001

0.001

0.01

0.1

1.0

Image 5
Image 6
Image 6
Image 6
Image 7
Image 8
Image 9
Image 6
Image 6
Image 10
Image 11
Image 12
Image 13
Image 6
Image 6
Image 14
Image 15
Image 16


Models

The results below are the comparisons of 3 images between the Vanilla GAN model and the StyleGAN model, both using a loss of 0.01 and the z latent space. It is pretty clear that StyleGAN results in significantly better results. The Vanilla GAN results in blurry, less detailed images. This is very likely due to StyleGAN being built with a neural style transfer architecture.

Vanilla GAN

StyleGAN

Image 5
Image 6
Image 6
Image 9
Image 6
Image 6
Image 13
Image 6
Image 6


Latent Spaces

The following are the results of StyleGAN with perceptual loss of 0.01 on different latent space initializations. Generally, speaking, I think the results for the W+ latent space are the best, with the excpetion of the top right image. It is evident that W+ preserves more of the fine grained detail (e.g. look at the bottom right image's background, and the middle right image's colors). This seems to align with the StyleGAN author's results as well. I believe that by briging the latent space closer to each of the generator's style convolutional blocks, the latent space is better to more effectively capture the high frequency details of the image.

Z

W

W+

Image 5
Image 6
Image 6
Image 6
Image 9
Image 6
Image 6
Image 10
Image 13
Image 6
Image 6
Image 14


Overall, I found that the optimal combination was a perceptual loss weight of 0.01, the StyleGAN architecture, and a latent space initialization of W+. With this combination, generating images takes roughly 2 minutes (on an M1 Mac) for 1000 iterations, which is plenty to get a good result.




Scribble To Image

Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2
Scribble Image 1 Scribble Image 2


These are the results of running the scribble to image pipeline with the parameters mentioned before. The last image is a sketch I created and the resulting cat image generated by the model. The first immediate observation I want to discuss is the difference in dense versus sparse sketches. Judging by these results, it seems like the model prefers sparse sketches a lot more. It seems that by giving the model more "free reigns" in filling in the gaps, it can provide much more realistic results, whereas when it has to try to match the denser sketches as closely as possible, it ends up looking too fake. It seems like the model also generally matches the sketch colors quite well.

Diffusion

Prompt = "Grumpy Cat reimagined as a royal painting."

N = 500, Strength = 15

N = 1000, Stength = 15

N = 1000, Strength = 30

Image 1
Image 1
Image 2
Image 3
Image 1
Image 4
Image 5
Image 6


Let's first take a look at the differences between running for 500 timesteps vs. 1000 timesteps. As shown in the SD edit paper, there is a sort of trade off between being faithful to the sketch and looking more realistic / being more faithful to the prompt. As we can see, running for only 500 timesteps gives us results incredibly similar to our sketch, however it lacks any sort of resemblance to a royal painting. On the flip side, when running for 1000 timesteps, we get something that truly resembles the prompt, while being less mindful of the sketch provided. I speculate this is due to the way this method works, where our initial latent isn't pure noise and is instead the sketch image. By denoising for fewer time steps, we can preserve the original input much better than running the algorithm for effectively twice as long.
Now, taking a look at the effect of doubling the strength of the CFG, it is not immediately clear what the effect is. From a brief glance, it seems to make the output more "sketch-like", where some of the realism is loss in favor for more colorful and cartoonish results. In theory, I believe that raising the strength of the CFG is supposed to make the model adhere more to the prompt. It is possible that the prompt, by not explicitly adding a word like "realistic", seems to generalize more toward sketch-like outputs (maybe also based on its training data).