16726-Learning Based Image Synthesis-Spring 24

Assignment #5 - Cats Photo Editing

Student Name: Peipei Zhong, Andrew ID: peipeiz

Overview

In this assignment, we need to manipulate images on the manifold of natural images by implementing a few different techniques.

In the first part, we will invert a pre-trained generator to find a latent variable that closely reconstructs the given real image.

Then we will use a hand-drawn sketch to generate an image that fits the sketch accordingly.

Finally, we will generate images based on an input image and a prompt using stable diffusion.

Bells & Whistles implemented: Experiment with high-res models of Grumpy Cat (2 pts)

Part 1: Inverting the Generator

Overview

First, I would like to clarify what is 'inverting a generator'. Generally, a GAN generator like Generator from the Vanilla GAN(shown as above) takes latent vector as input and generate images.

But in this case, we are given the pre-trained generator and the input image, and we need to find the best latent code that can generate the image that is most similar to the input image. Since this process is the reverse of what the generator does, we call it 'inverting the generator'.

z^∗=arg min_z L(G(z),x)

I use the LBFGS solver, which is adept at handling non-convex optimization issues, to solve this optimization problem.

Results

Target cat:

$L(G(z),x)=\lambda_{perc}L_{perc}+w_{L1}L_1$

$L(G(z),x)=\lambda_{perc}L_{perc}+w_{L2}L_2$

$\lambda_{perc}$ $w_{L1}$ $w_{L2}$ are weights.

In all the experiments, the iteration is 1500. L1 weight is fixed to 10, and L2 weight is fixed to 0.01.

vanilla, latent z $\lambda_{perc}$

stylegan, latent z $\lambda_{perc}$

stylegan, latent w $\lambda_{perc}$

stylegan, latent w+ $\lambda_{perc}$

The outputs of stylegan have better quality and higher resolution than vanilla, and have more detailed and sharper edges. This is because stylegan is a more complex and larger model.

And for the same gan model, the output is very similar, only very subtle difference, like the color of the fur, and the shape of the eye. And latent w+ has better output than z and w. We can see that the lower right corner of the cat’s face is slanted rather than rounded, w+ restores this better. This may because w+ integrate different w vectors for different layers, which are enhanced by AdaIn's layer-specific adjustments.

Based on the above experiments, I think using StyleGANw+ $\lambda_{perc}$ $\lambda_{perc}$ = 0.1 also has good output. This is because the shape of the face and eyes are most similar.

$\lambda_{perc}$ = 0.0 and use L2 loss, the output performance is really bad, which means only L2 loss can not find good latent vector.

Running time: Using a single 4090 GPU, the vanilla GAN around takes 6 seconds to run, while styleGAN's runtime is around 18 seconds.

Part 2: Scribble to Image

In this part we still manipulate image in the latent space. The main idea here is we use a mask that come from scribble to indicate which pixels need to be considered. And the loss is :

z^* = arg min_z||M\circ G(z) - M\circ S||_1

$\circ$ is the Hadamard product, M is the mask, and S is the sketch.

$\lambda_{perc}$ = 0.01, L1 loss=10, iteration is 1500.

We can see that with sparse sketches, the output is bad and blur. This may because too little silhouettes can not provide enough information for the model to optimizing.

With dense sketches, we have better output. Adding more details and use consistent color can output images that match better with the sketch. Like the result of line 4, given blue eyes in the scrible, the output also have blue eyes, which is consistent with Grumpy cat.

Part 3: Stable Diffusion

In this part, we nee to use Stable Diffusion to synthesizes images by incorporating an input image along with a text prompt, which is similar to SDEdit. And we need to extend SDEdit with a text-to-image Diffusion model, using the DDPM sampling method with the Classifier-free Diffusion Guidance.

The results here are generated using default hyper-parameters:

guidance strength = 15, timestep = 500, noise seed = 10, Prompt: "Grumpy cat reimagined as a royal painting"

The output looks similar to the input, but I think it has nothing to do with the input prompt " royal painting".

Experiment on different timestep:

Fix strength = 15 and noise seed = 10

We can see that with 999 iteraton, the output have a photo frame, which is match with the "royal painting" in the prompt, but the cat in it is not exactly the same as the input, but it is still a grumpy cat and it is an abstraction of the input. With timestep become larger, the more noise will be added, so it would generate more diverse results.

Experiment on different noise seed:

Fix strength = 15 and timestep = 500:

With different seed, we would get different output, so we can try several seed until we get the output we like.

Experiment on different classifier-free guidance strength values

Fix noise seed = 10 and timestep = 500:

Small strength values would lead to simple output image, and large strength values would lead to more complicated and detailed results.

Bells & Whistles

I use cat images with 256*256 as input:

parameter: use style gan, --perc_wgt 0.01, use l1 loss=10, iteration = 1000

Here are some results:

We can have much better results compare to use low resolution images as input. I think this is because high resolution images can provide more details for the model.

The performance: w+ > w > z.

Scribble	Mask	Output