Cats Photo Editing

Brief introduction

In this assignment, I implemeted a few different techniques that require to manipulate images on the manifold of natural images. The techniques include Vanilla GAN, StyleGAN, and stable_diffusion. In the first part, I reconstructed input images with a reconstruction loss using Vanilla GAN and StyleGAN. In the following section, I explored using a scribble to guide the reconstruction process. In the last part, I implemeted the inference of the stable diffusion model and added a image guidance to the process in order to get diverse but realistic samples.

Inverting the Generator

The problem could be formulated as follows: given a generator \(G\) and a target image \(x\), we want to find a latent code \(z\) such that \(G(z) \approx x\). The loss function is defined as the l1 loss between the generated image and the target image. Specifically, I used LBFGS optimizer to optimize the latent code. In this section, I only used perceptual loss and pixel loss. The weight of the pixel loss is fixed to 10.

Experiment on losses

As described above, the loss function is defined as:

\(L(G(z), x) = \omega_{\text{perc}} \cdot L_{\text{perc}}(G(z), x) + \omega_{\text{pix}} \cdot L_{\text{pix}}(G(z), x)\)

The perceptual loss comes from the previous assignment, and is defined on conv_5 here. I use StyleGAN model and z latent space across all experiments. Here are the results from different weights of perceptual loss.

Overall, some results from 0.01 and 0.001 preserved the light, shape, and color best, but there are not many differences. I personally picked weight as 0.01 for the following experiments.

Experiment on different generative models

I tried two pretrained generative models, Vanilla GAN and StyleGAN using z latent space.

We can see that StyleGAN outperforms Vanilla GAN in terms of preserving the global and local structures. Vanilla GAN generate more noise and mosaics. This is because StyleGAN improves the reconstruction quality by enhancing the smoothness of the latent space, introducing a multi-scale generator, and implementing better feature normalization and control mechanisms.

Experiment on different latent spaces

I tried three latent spaces, z, w and w+. These are terms for StyleGAN. z represents random vectors sampled from a standard normal distribution, serving as input to the generator network. The w space, derived from z, enables fine-grained control over image style. w+ extends w to support conditional image generation, offering greater flexibility and control. The key difference between w and w+ is that w+ injects different latent code into network layers while w space uses the same one. Here are my ablation results.

We can see that results from w+ spaces are the best. This is because w+ vector allows for more fine-grained control across different network layers comparing with w spaces, so that the model could better capture various features and variations in the data. Also, learning meaningful representations directly from z is challenging since there are less controls. The model may struggle to generate images with desired characteristics or styles, resulting in suboptimal performance.

Speed to run

I found that Vanilla GAN is much faster than StyleGAN. It takes around 24 seconds for StyleGAN to optimize 1000 iterations, while the same number of iterations only takes ~7 seconds for Vanilla GAN. This is reasonable because StyleGAN is more complex.

Scribble to Image

The main idea of this section is very simillar to part 1. Instead of reconstructing the whole image pixel by pixel, we have a mask for each scribble image, and we only need to apply perceptual loss and pixel loss on the restricted area. Now our objective becomes:

\( z^* = \arg \min_z \| M \circ G(z) - M \circ S \|_1 \)

Note - about hyper-parameters: I introduced l2 regularization to avoid overfitting and improve the output quality. The weight of regularization is 0.003. Use --lambda_reg to control the weight. The weight of perceptual loss and pixel loss are both 0.5. According to my exploration in Part 1, I would use w+ latent space together with StyleGAN to generate results.

Unfortunately, the results are not as good as I expected. This might happen because the input guidance is not detailed enough to generate vivid images. Looking into details, we could find that row 4&5 looks relatively best, since the corresponding scribbles use large chunks of color rather than simple lines. Row 1 to 3 generate random results that seemingly have little to do with our input for using coarse lines that offer inadequate constraints. Note that row 2 didn't optimize at all due to lack of guidance.

The last 4 rows have something wrong on their edges. I personally think it is because some scribbles don't look like real grumpy cats, and some applied color that is not in the original image. On the contray, the scribbles used bright blue in the cat's eyes helped generate rather good results.

Stable Diffusion

The Diffusion model is a text-to-image generative model that can generate high-quality images. Since depending only on text input may generate unexpected results, we would like to add image guidance to the process. The main idea in the algorithm is to add noises into the input image to initialize, and then iterate reversely by the timestep to denoise, eventually get the output image guided by the input prompt.

Experiment on different input images

Besides the given scribble, I also tried another image found on the Internet. The results here are generated using default hyper-parameters, i.e. guidance strength = 15, timestep = 500, noise seed = 10.

Experiment on adjusting the noise

The results from last experiment looks not that cool. So I first explored the best timestep to use. Since the larger the timestep is, the more noise will be added, setting a larger timestep would generate more diverse results.

I prefer the results from 700 iterations. For 900 iteration, the outputs have limited correlations with the original input, except that the position that the cat's face is looking to. While for 500 iteration, it still mostly resembles the input image and doesn't look like a 'royal painting' as requested by our prompt.

I also tested two different random seeds(using 700 iterations). Here are the results.

The outputs are so different! I learn from this experiment that we could generate multiple results at a time using different seeds since we don't know which one we would prefer the most.

Experiment on different classifier-free guidance strength values

I tried 3 different guidance strength values, 5, 15, and 25. The results with 15 are the best. I noticed that the strength cannot be too small or too large, otherwise some geometry might be misintepreted by the model. When the strength is too small, the guidance is not adequate for the model to generate an image that looks like the input, while when it's too large, the model tries too hard to make the differences between conditional and unconditional results and make mistakes.