**Learning-Based Image Synthesis** **Assignment 5** - Cats Photo Editing Overview ============================================================== The goal of this assignment is to implement a few different techniques that manipulates images on the manifold of natural images. First, we will invert a pre-trained generator to find a latent variable that closely reconstructs the given real image. In the second part of the assignment, we will take a hand-drawn sketch and generate an image that fits the sketch accordingly. Finally, we will generate images based on an input image and a text prompt using stable diffusion. Inverting the Generator ============================================================== Here we solve a non-convex optimization problem to reconstruct the image from a latent code. Ablation - generative models & latent space --------------------------------------------------------------  | Model, latent-space | Output | |----------|---------| | vanilla GAN, z | | | styleGAN, z | | | styleGAN, w | | | styleGAN, w+ | | Reconstructions from styleGAN is more detailed than vanilla GAN and particularly using the w+ space gives more sharper images with more clarity than w and z spaces. Some of the reconstructions are grumpy cat though the target image isn't a grumpy cat because the dataset on which it is trained on contained mostly grumpy cats. Ablation - losses -------------------------------------------------------------- We use a combination of L1 and perceptual loss (content loss after the 5th convolutional layer in a pre-trained VGG-19 network) in the optimization procedure.
Loss = L1_wgt * L1_loss(G(z), x) + Lperc_wgt * L_perc(G(z), x)
I keep L1_wgt as 10.0 and vary Lperc_wgt using a StyleGAN on its z latent space.  | Lperc_wgt | Output image | |----------|---------| | 0 | | | 0.1 | | | 0.01 | | | 0.001 | | | 0.0001 | | From the above visualizations, it seems that the weight for perceptual loss shouldn't be too less or too large. 0.01 seems to be ideal. I did try some comparisons with L2 loss but it seemed to be doing more harm than good. Hence, I didn't go further with L2 loss. For L1 loss, L1_wgt as 10.0 seems to be ideal.

All the above configs run very fast on an A100 GPU. StyleGAN takes more time than vanilla GAN. StyleGAN takes around 1.5 minutes whereas vanilla GAN takes less than 30 seconds for 1000 iterations. There is no significant difference in running time between different latent space reconstructions.

It should be noted here that many reconstructions are of grumpy cat but the target image isn't grumpy. I think this is to do with the bias in the training data for grumpy cats and maybe it won't happen so much with hyper-parameters optimization. Scribble to Image ============================================================== For converting a scribble of a cat into a realistic looking cat, we solve it as a penalized nonconvex optimization problem. Along with the above optimization we add a color constraint using a mask for each input image. This ensures that wherever there's colour in the input image, those pixels act as target colour for the reconstructed image. | Input sketch | Output image | |----------|---------| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Denser sketches seem to give more realistic output while enforcing the dense color constraints. The sparser sketches aren't that bad though, only thing is the sparse colours permeate in other places of the cat which is evident in the 2nd and 7th cat. In the denser sketches like 4 and 5, the more realistic the sketch is, more realistic is the output image. 10th cat is an interesting observation. I wanted the reconstructed cat to have red ears and green lips but it can be seen that the red colour has been permeated across in other regions. Overall the colour constraints seem to be working well. Stable Diffusion ============================================================== In this section, we use an approach similar to SDEdit to generate images based on a text prompt and an input image using a pretrained Stable Diffusion model. Ablations -------------------------------------------------------------- Input image

Input prompt - "Grumpy cat reimagined as a royal painting" - Different seed | seed = 1 | seed = 10 | seed = 100 | seed = 1000 | |----------|---------|---------|---------| | | | | | - Different iterations | iters = 50 | iters = 250 | iters = 500 | iters = 800 | |----------|---------|---------|---------| | | | | | - Different classifier-free guidance strength values | strength = 5 | strength = 15 | strength = 30 | strength = 100 | |----------|---------|---------|---------| | | | | | I use seed = 10, iterations = 800, strength = 15 for the further results. Results -------------------------------------------------------------- | Input prompt | Input image | Output image | |----------|---------|---------| | "A fantasy landscape with a river in the middle and a castle in the corner" | | | | "interiors of a rolls-royce car" | | | | "a realistic horror photo of a skull with scary teeth" | | | | "a gladiator charging for war with a sword in a sunny desert" | | | The 1st input image was taken from the previous year's homework. The 2nd and 3rd input image was taken from https://www.reddit.com/r/StableDiffusion/ and the 4th input image is my poor drawing on laptop browser using touchpad. Bells & Whistles ============================================================== Interpolation between two latent codes --------------------------------------------------------------