This project aims to use deep learning techniques to manipulate images on the manifold of natural images. The project has following 3 parts:
Part 1: Inverting the Generator
This part explores a way to reconstruct the image (finding an optimal z*) from a particular latent code z via solving an optimization problem. The nonconvex optimization problem is shown below. Where loss is L and G refers to the generater. x refers to the original image, and z is the latent vector. In this part, I use the Lp loss (typically L1 loss), perceptual loss and L2 regularization for optimization (LBFGS optimizer).
Part 2: Scribble to Image
This part's goal is to constrain the sythesized image to make it look more realistic. I use color scribble constraints in this problem.
Part 3: Stable Diffusion
In this part, I implement DDPM sampling method to utilize stable diffusion model for synthesizing images from text prompts. The algorithm is shown below.
Experiment setup:
Random Seed: 725
Iterations: 1000
Perceptual Loss: Use feature from the output of the 5th block of the VGG-19 model.
Target Image:
Platform: WSL2-Ubuntu22.04 LTS, I9-12950HX, 3080 Ti Laptop GPU GDDR6
Experiment 1 ablation study for l2 norm: Fixed Perceptual loss weight 0.01, fixed Latent code: w+ (from mean), fixed Generate model (StyleGAN) and fixed Lp (L1) Loss weight: 10. L2 norm of delta weight (regularization): 0, 0.001, 0.01, 0.1.
Discussion:
We could see that the smaller the L2 norm weight is the more similar the generated image will be compared to the target image, and the larger the L2 norm weight is, the more similar the generated image will be compared to the image synthesized from random noise by StyleGAN (since the image generated with l2 norm weight 0.1 looks really similar to the one with 0 l1 weight in the next experiment). Since the L2 norm tend to reduce the magnitude of the delta, which greatly reduces the effect of the optimization process.
Experiment 2 ablation study for Lp (L1) loss: Fixed Perceptual loss weight 0.01, fixed Latent code: w+ (from mean), fixed Generate model (StyleGAN) and fixed L2 norm of delta weight (regularization) 0.001. Varying L1 loss: 0, 1, 10, 100
Discussion:
We could see that the larger the l1 loss weight, the more similar the generated image is compared to the target image. Since L1 loss aims to make the generated image identical to the target image, thus, the larger the L1 loss weight the more similar the generated image is compared to the target image.
Experiment 3 ablation study for Perceptual loss: Fixed Lp (L1) loss: 10, fixed Latent code: w+ (from mean), fixed Generate model (StyleGAN) and fixed L2 norm of delta weight (regularization) 0.001. Varying Perceptual Loss weight: 0, 0.01, 0.1, 1
Discussion:
From the generated image, we could see that the larger the perceptual loss weight, the less simialr the image compared to the target image. Moreover, the larger the perceptual loss weight, the less detail (especially color information) the generated image has. It is intuitive, since the perceptual loss is calculated based on the output of the fifth block of the VGG-19 model, and the module here mainly keeps the overall layout information (with little color information). Moreover, the larger the perceptual weight is, the less focus the optimization will do to make the image looks the same to the target (since l1 loss's ratio is reduced).
Experiment 4 ablation study for Generative models: Fixed Lp (L1) loss: 10, fixed Latent code: z, fixed Perceptual loss weight 0.01 and fixed L2 norm of delta weight (regularization) 0.001. Varying Generative model: Vanilla GAN and StyleGAN, from mean or not from mean
Discussion:
From the generated image, we could learn that the images generated by StyleGAN has much more details than the images generated by Vanilla GAN (and Vanilla GAN cannot generate anything if given latent code as all 0s: z mean). This makes sense, since the StyleGAN training strategy allows the model to generate images with more details.
Experiment 5 ablation study for latent code: Fixed Lp (L1) loss: 10, fixed Generative model: StyleGAN, fixed Perceptual loss weight 0.01 and fixed L2 norm of delta weight (regularization) 0.0001. Varying latent code: z, z from mean, w, w from mean, w+, w+ from mean
Discussion:
From the generated image, we could see that from mean performs better than not from mean, and w+ performs the best. I think the reason for this that taking the mean requires the model the generate a lot more times than not taking the mean. This could help reduce noise in the latent code. And for the w+, it performs better because it keeps more information (nw other than 1).
The best hyper-param: StyleGAN, w+ from mean, Perceptual weight 0.01, L2 regularization weight 1e-3/1e-4 (it depends), L1 loss 10.
For the average running time:
StyleGAN: around 29s
Vanilla: around 14s
Experiment setup:
Random Seed: 725
Iterations: 1000
Perceptual Loss: Use feature from the output of the 5th block of the VGG-19 model.
StyleGAN, w+ from mean, Perceptual weight 0.01, L2 regularization weight 0.0, L1 loss 10.
Discussion:
From the results we could get to the conclusion that:
The more or larger color blocks provided in the sketch (more dense), the more likely that the generated images will have less details in these color blocks (affected by l1 loss, since the color blocks may increase the l1 loss while providing no more perceptual information(not increasing the perceptual loss)). Thus, the generated images will tend to have more color blocks like the sketch and less details (since color blocks lack details).
Sparse sketch can help slightly modify the layout of the object in the generated images.
How dense/sparse the sketch should be also related to the hyper-param. For example, the following images is generated with perceptual loss 5, given the same mask, the generated image's quality is very different.
Experiment 1 ablation study for amounts of noise added: 300, 400, 500, 600, 700 steps (strength 70)
Image 1: Prompt: jojo style character
Image 2: Prompt: Grumpy cat reimagined as a royal painting
Discussion:
We could see that with more noise added, the generated image becomes closer to our prompts with fixed classifier-free guidance strength value. Since more noise added means more denoising steps (just like training a neural network, more training steps usually leads to a network that better fits the training data), which can help the algorithm to change the image more and make it closer to what we described in the prompt.
Experiment 2 ablation study for classifier-free guidance strength values: 10, 30, 50, 70, 90 (500 steps)
Image 1: Prompt: jojo style character
Image 2: Prompt: Grumpy cat reimagined as a royal painting
Discussion:
We could see that with larger strength value, the generated image becomes closer to our prompts with fixed noise amount. Since larger strength value will bring larger impact to the image, but larger strength value does not always mean faster and better quality, for example, the one with guidance strength value 90 (a lot of noise and artifacts). (just like training a neural network, larger learning rate usually makes the network converges faster, but larger learning rate may lead to poorer performance as well)