Assignment 5: Cats Photo Editing | 16-726: Learning-Based Image Synthesis | prabhdees

Bells & Whistles attempted

0. Overview

In this assignment, we focus on manipulating images on the manifold of natural images using various techniques. Initially, we invert a pre-trained generator to find a latent variable that reconstructs a given real image accurately. Subsequently, we generate images from hand-drawn sketches and create further images based on these sketches and textual prompts using stable diffusion. Throughout, we conduct extensive experiments with different models, loss functions, and hyperparameters to optimize each task and analyze the impact on image generation.

1. Inverting the Generator

In this part, we address the challenge of reconstructing an input image by inverting a pre-trained generator to optimize a latent variable. This process involves minimizing the loss between the generated and input images using various models, loss functions, and hyperparameters to identify the most effective combination. Specifically, we employ the perceptual loss metric, utilizing the conv_5 layer of the VGG-19 network to calculate the content distance by comparing features extracted from both images. This task is framed as a nonconvex optimization problem, grounded in the concept that natural images reside on a low-dimensional manifold closely approximated by the output manifold of a trained generator.

1.1 Loss Variations

Experiment Setup: L1_wgt: 10, L2_wgt: 0, LPerc_wgt: varying
| Target | LPerc_wgt: 0.1 | LPerc_wgt: 0.01 | LPerc_wgt: 0.001 |:-:|:-:|:-:|:-:| | | | | |

Experiment Setup: L1_wgt: 0, L2_wgt: 10, LPerc_wgt: varying
| Target | LPerc_wgt: 0.1 | LPerc_wgt: 0.01 | LPerc_wgt: 0.001 |:-:|:-:|:-:|:-:| | | | | |

From the above results it is clear with our small ablation that both l2 and l1 loss with weight 10 and with Perceptual Loss with weight 0.01 works the best.

1.2 Model Variation

Experiment Setup: L1_wgt: 10, L2_wgt: 0, LPerc_wgt: 0.1
| Target | Vanilla | StyleGAN |:-:|:-:|:-:| | | | |

From the above results it is clear that StyleGAN outperforms Vanilla GAN

1.3 Latent Space Variation in StyleGAN

Experiment Setup: L1_wgt: 10, L2_wgt: 0, LPerc_wgt: 0.1
| Target | z | w | w+ |:-:|:-:|:-:|:-:| | | | | |

From the above results we can see that w performs the best for 1000 iters. However it is important to note that as we increase the number of epochs w+ results in best results.

1.4 Best Hyperparameters

Finally the best performing hyperparameters are:

L1_wgt: 10
L2_wgt: 0
LPerc_wgt: 0.01
Model: StyleGAN
Latent Space: w
n_iters: 1000

2. Scribble to Image

Experiment Setup: Used Best Hyperparameters from 1.4
| Sketch | Mask | Image | |:-:|:-:|:-:| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

Experiment Setup: Used Best Hyperparameters from 1.4
| Sketch | Mask | Image | |:-:|:-:|:-:| | | | | | | | |

Please excuse my sketch I'm real bad at it but glad that the model is robust to such bad sketches like in 1

Insights: The experimental results indicate that using sparse masks tends to produce better-detailed images, as these allow for greater flexibility during optimization. Conversely, denser masks lead to images that closely resemble the input sketch due to the optimizer adjusting a larger number of pixels, which often results in an over-constrained generation process. Furthermore, excessively dense masks that deviate significantly from the space represented by the generator can cause the optimizer to fail to converge, resulting in images that do not effectively mimic the sketch.

3. Stable Diffusion

In this task, we adapt the SDEdit approach where the input image acts as a "guide" by initiating the diffusion process with this image instead of random noise, subsequently transforming it into noise through the forward diffusion process. This is then iteratively denoised to generate realistic images using a pre-trained diffusion model, specifically employing the DDPM sampling method with Classifier-free Diffusion Guidance. Unlike typical applications of Stable Diffusion that generate images purely from textual prompts, this modified approach integrates both an input image and a text prompt to guide the image synthesis. This extension of the SDEdit method allows for more controlled and contextually relevant image generation without using the DDIM sampler.