**Cats Photo Editing** # Overview This assignment focuses on image editing techniques that involve manipulating images within the realm of natural images. The first part involves inverting a pre-trained generator to find a latent variable that closely reconstructs a given real image. In the second part, you will take a hand-drawn sketch and generate an image that matches the sketch. The final part involves generating images based on an input sketch image and a prompt using stable diffusion. Throughout these tasks, you will experiment extensively with various models, loss functions, and hyperparameters to identify the best combination for each task and observe their effects on the generated images. # Inverting the generator In this task, you will utilize a pre-trained generator and invert it to reconstruct an input image. The process involves optimizing the latent variable to minimize the loss between the generated image and the input image. You will experiment with different models, loss functions, and hyperparameters to identify the best combination for this specific task. All the experiments in this section ran for a total of 1000 iterations, which took approximately 30 seconds to complete on a single 2080 Ti GPU. ## Ablation across different losses (latent fixed to w+, from_mean = True) We use a combination of L1 and perceptual loss over the generated and L2 regularization loss on delta on the parameter with different weights to choose the best hyperparameters

From the above table we deduce the best values of the hyper-parameters to be : perceptual loss weight = 0.1, L1 loss weight = 10.0, L2 regularization weight = 0.0.

Perceptual loss weight	L1 loss weight	L2 regularization weight	Generated Image
0.01	10.00	0.0
0.1	10.00	0.0
0.01	10.00	0.01
0.01	5.00	0.0
0.1	10.00	0.01

As expected, the perceptual loss weight has a significant impact on the generated image and helps in preserving global level features. However, some pixel based reconstruction loss is required to actually generate a sharper image. Hence, L1 loss helps. However, the L2 regularization on delta just seems to make the optimization slower. ## Ablation across 'from_mean' (best loss weights taken from loss ablation above)

from_mean	Generated image
True
False

From the above table we deduce the best values of the hyper-parameters to be `from_mean=True` Using sampling from_mean helps in more stable optimization and hence better results. ## Ablation across 'latent' (best loss weights and from_mean taken from ablations above)

latent	Generated image
w+
w
z

The table illustrates the impact of different latent space representations on the quality of reconstructed images. The most effective approach appears to be the utilization of the 'w+' latent space, which yields superior results compared to the alternatives. This can be attributed to the enhanced capacity of the 'w+' latent space to encode comprehensive information about the input image. In contrast, the 'z' latent space, which is associated with the vanilla GAN architecture, encodes relatively limited information about the image, leading to suboptimal results. The 'w' latent space offers an improvement over 'z' by encoding more image information, but it still falls short of the performance achieved with the 'w+' latent space. Consequently, the table suggests that the optimal choice of hyperparameters involves the adoption of the 'w+' latent space representation, as it facilitates the most effective encoding of image. ## Ablation across models (best loss weights, from_mean, and latent taken from ablations above)

model	Generated image
StyleGAN
vanilla GAN

The table indicates that the StyleGAN model, with its more expressive and information-rich latent space representation, outperforms the vanilla GAN architecture in generating high-quality images, making it the preferred choice for image generation tasks. # Scribble to Image

Sketch	Mask	Generated Image

The results demonstrate that sparse masks yield superior outcomes compared to denser masks. When a denser mask is employed, the generated image undergoes more drastic changes from the latent representation, causing the final image to closely resemble the input sketch. This effect can be attributed to the optimization process operating on a significantly larger number of pixels in the case of a dense mask, as opposed to a sparse mask. Furthermore, it is evident that when the mask becomes excessively dense, the optimization becomes way more contraint creating very less deviation of the generated image from the sketch. Shown below are some results on my own sketches

Sketch	Mask	Generated Image

We can see despite my terrible sketches, the generated images try to replicate the cat with the dominant color coming from the sketch. The last example depicts a failure case as a red cat falls out of the distribution of the GAN. # Stable Diffusion In this task, we adopt an approach analogous to SDEdit, where the input image acts as a guiding reference. Instead of initiating with random sampling and subsequently generating a realistic image through iterative denoising using a pre-trained diffusion model, we transform the provided input image into noise via the forward diffusion process. Thereafter, we employ the DDPM sampler to iteratively denoise the noisy input, effectively reconstructing a realistic image while adhering to the guidance offered by the initial input image.

The results below are all shown on the example input image.

Results across prompts. We can see that the change in prompt does not significantly change the output image as the guidance image is the same.

Prompt	Generated Image
Grumpy cat reimagined as a royal painting
Happy cat reimagined as a royal painting

Results across seed values. Changing the seed changes the initialization latent and results slight tweaks to the images

seed	Generated Image
10
0
5

Results across guidance strengths. Very guidance strength results in missing features in the generated image.

Guidance strength	Generated Image
15
5
30

Results across ddpm sampling steps

DDPM sampling steps	Generated Image
500
800

Results on some drawings and sketches. Here I show the results using two rudimentary crayon drawings and one of my cat sketches

Prompt	Input Image	Generated Image
A house on a field with a road leading up to it on a sunny day
A beautiful road leading to snow covered mountains through a jungle
Orange Cat smiling

# Bells and Whistles ## Interpolation between two latents

Image 1	Interpolation gif	Image 2