Learning Based Image Synthesis (16-726)

Assignment 5

Aviral Agrawal (avirala)

Bells & Whistles attempted

1. Overview

This assignment revolves around image editing by manipulating images on the manifold of natural images. First, we will invert a pre-trained generator to find a latent variable that closely reconstructs the given real image. In the second part of the assignment, we will take a hand-drawn sketch and generate an image that fits the sketch accordingly. Finally, we will generate images based on an input sketch image and a prompt using stable diffusion. For all these tasks, we experiment extensively across various models, loss functions, and hyperparameters to find the best combination for each task and observe their effects on the generated images.

2. Inverting the Generator

For this task, we use a pre-trained generator and invert it so as to reconstruct an input image. We optimize the latent variable to minimize the loss between the generated image and the input image. We experiment with different models, loss functions, and hyperparameters to find the best combination for this task.
All the experiments in this section ran in total for 10000 iterations in around 3 minutes on a single A6000 GPU.
2.1. Ablation across different losses (latent fixed to w+, from_mean = True)
We use a combination of L1, L2, and perceptual loss with different weights to choose the best weighting.

Perceptual loss weight	L1 loss weight	L2 loss weight	Best Loss	Best Iteration	Generated Image
0.001	0	0	0.111	3750
0.01	0	0	0.333	7750
0.5	0	0	17.993	3250
5.0	0	0	180.567	9000
10.0	0	0	377.789	3750
0.01	10.0	0	1.233	8750
0.01	5.0	0	0.788	9750
0.01	1.0	0	0.414	10000
0.01	5.0	10.0	0.907	8250
0.01	5.0	5.0	0.863	8750
0.01	5.0	1.0	0.765	9750

From the above table we deduce the best values of the hyper-parameters to be : perceptual loss weight = 0.01, L1 loss weight = 5.0, L2 loss weight = 0.0.
As expected, the perceptual loss weight has a significant impact on the generated image and helps in preserving global level features. However, some pixel based reconstruction loss is required to actually generate a sharper image. Hence, L1 loss helps. However, L2 loss is too strict a penalty and we begin loosing the global picture by using L2 loss.
2.2. Ablation across 'from_mean' (best loss weights taken from loss ablation above)

from_mean	Best Loss	Best Iteration	Generated Image
True	0.788	9750
False	0.838	9250

From the above table we deduce the best values of the hyper-parameters to be : from_mean = True.
Using sampling from_mean helps in more stable optimization and hence better results.
2.3. Ablation across 'latent' (best loss weights and from_mean taken from ablations above)

Latent	Best Loss	Best Iteration	Generated Image
z	1.619	7750
w	1.072	9000
w+	0.788	9750

From the above table we deduce the best values of the hyper-parameters to be : latent = w+.
Using a simple 'z' latent space, we are not encoding much information about the image since it uses vanilla GAN. Using 'w' latent space, we are able to encode more information about the image and hence get better results. Using 'w+' latent space, we are able to encode the most information about the image and hence get the best results.
2.4. Ablation across models (best loss weights, from_mean, and latent taken from ablations above)

Model	Best Loss	Best Iteration	Generated Image
Vanilla GAN	1.619	7750
StyleGAN	0.788	9750

From the above table we deduce the best choice of model to be : model = StyleGAN.
StyleGAN latent space is much more expressive and encodes much more information than vanilla GAN. Hence, we get better results using StyleGAN.

3. Scribble to Image

In this task, we take a hand-drawn sketch and generate an image that fits the sketch accordingly.
3.1. Results on sketches provided with the assignment

Sketch Image	Mask Image	Generated Image

From the above results we see that sparse masks give better results. With a denser mask, the generated image from the latent changes much more dramatically and the final image tends to look very much like the input sketch. This is the optimization is now running over much more pixels than in the case of a sparse mask.
Also, we can see that when the mask is way too dense and different from space represented by the generator, the optimizer fails to converge and the image does not mimic the sketch.
3.2. Results on my own sketches

Sketch Image	Mask Image	Generated Image

With these images also, we observe the same trend as above.

4. Stable Diffusion

In this task, we create an approach similar to SDEdit, in which the image input serves as a “guide,” transforming the given input image into noise through the forward diffusion process instead of starting with random sampling and then iteratively denoising to generate a realistic image using a pre-trained diffusion model. We use DDPM sampler.

4.1. Results across prompts

Prompt	Input Image	Generated Image
"Grumpy cat reimagined as a royal painting"
"Grumpy cat reimagined in a mona lisa style painting"

The prompt for all the following images is : "Grumpy cat reimagined as a royal painting"

4.2. Results across input images

Input Image	Generated Image

4.3. Results across different seed values

Seed	Input Image	Generated Image
10
20

4.3. Results across different iteration values

Num Iterations	Input Image	Generated Image
100
500
900

4.3. Results across different guidance strength values

Guidance Strength	Input Image	Generated Image
11
15
100

5. Bells & Whistles : Interpolate between two latent codes

We interpolate latent space to generate a gif.