This assignment focuses on cat photo editing, by manipulating the manifold of images. There are 3 parts: Inverting the Generator, Scribble to Image, and Stable Diffusion.
Part 1: Inverting the Generator
The first part of this assignment focus on reconstructing an image from a latent representation. We get our latent representation from a trained generator, the generator output is similar enough to the input image's manifold to work as our latent representation.
With our generator, G, and input image, x, we can formulate z*, the output image, with loss as follows.
We use 3 losses: L1, L2 and content loss after the 5th convolutional layer.
We use three different versions of latent code spaces, z, w and w+. We also have 2 different generators: GAN (vanilla) & StyleGAN.
Different Latent Codes
Lets take a look at what our reconstruction images look like when all the losses are weighted equally (weight = 1), but we vary the latent space type.
When using the vanilla generator our results are not great, using StyleGAN in z space gives us about the same results loss wise, but visually better. When we use styleGAN with w and w+ space the results start looking very good.
StyleGAN does a little better than the GAN (vanilla) implementation initially, but it does provide us the ability to use different intermediate latent spaces like w & w+.
As mentioned in the StyleGAN paper, w (an intermediate latent space) may not be enough to get improved results, where w+ (a concatenation of several w vectors, 1 for each layer of StyleGAN) is more suited for the architecture and provides substantial improvement in results.
The vanilla method is faster than StyleGAN. The vanilla method took about 40 seconds on my system, and StyleGAN took 2 minutes and 20 seconds.
Note that the first cat has a lower loss with w+ but visually the results look poorer. This is a one-off case due to the difficulty of the image. It seems, due to the dataset mostly being grumpy cats, and backgrounds being hard to reconstruct, this particular cat is very difficult. This cat image has a complex floral background and is not a "grumpy cat" breed.
Different Weight Combos
Now that we have evidence StyleGAN with a w+ latent space performs the best, lets try different weight parameters on our losses to improve the performance of this model.
Using StyleGAN & w+, we leave two of the weights at 0.1 and the third weight is 10, as shown below. This simple test shows that L1 loss is the most important.
L1 loss needs to be weighted the highest for the best results. If the weight of the perceptual loss is too high, the results risk becoming too unrelated to the input. Weighting L2 loss higher than the others is not as good as putting more importance on L1 loss.
Part 2: Scribble to Image
The next task is to take an input scribble of a cat and output a more realistic looking cat. These scribbles place a constraint on our desire output image, in our case this constraint is color. If the sketched cat has blue eyes, we want the output cat to have blue coloring in the same area, hopefully giving the output cat blue eyes as well.
Similar to the last problem we have a generator z, but now we have 2 input images. M, the mask, which tells us which pixels in our second input, S, the sketch, to constrain our output with.
z* is now formulated as the following
This equation translates to wanting a given pixel in our output image to match the color at the same location in the sketch image, if the mask is true at said pixel.
Results
Using our best model from part 1, with StyleGAN, w+, and weights of L1 = 10, L2=0.1 and prec = 0.1, we changed our z* function to the above and got the following results.
The model seems to do better with denser sketches. It seems when the model is not given enough color guidance, enough constraint, the blanks get filled in, in a less than ideal way. For example look at the first and third cats. They have more of a blurry effect than the other two cats, and overall look less realistic.
When not given a color to match at a pixel, the new color seems to be based on the surrounding pixels, which may not resemble a real cat, causing our blurry results. This is seen best with the excess of pink in the first cat.
Part 3: Stable Diffusion
The final part of this assignment branches away from teh first two and focuses on stable diffusion. We modify stable diffusion to take an image input as well as text input.
The approach we take is based off of SDEdit. The algorithm we implemented is as follows
Results
Let us first look at results with the given image and prompt. The default parameters were used to get the below results. The seed was 10, with 1000 time steps and a strength of 15.
Differing Strengths
With our number of time steps at 1000, and seed of 10, lets vary the strength. The strength applies to the guidance value.
The higher the strength, the more complex our output is.
Differing Time Steps
With our strength back at 15, and seed of 10, lets vary the number of time steps.
The higher the number of time steps the more our result diverges from the input image.
Differing Random Seed
With our number of time steps at 1000, and strength of 10, lets vary the seed. The seed will change our initialization, resulting in a different output from other seeds.
