Overview

For this assignment, we delve into image editing techniques specifically tailored for natural images. Initially, we invert a pre-trained generator to uncover a latent variable that accurately reconstructs a given real image. Next, we create an image based on a hand-drawn sketch. Finally, we explore image generation using an input sketch and a prompt via stable diffusion. Throughout these tasks, we experiment with various models, loss functions, and hyperparameters to find the optimal combination for each task and observe their impact on the resulting images.

GAN Inversion For some choice of loss L and trained generator G and a given real image x, we can write

$$z^{\ast }=\arg \underset{z}{min}\mathcal{L}(G(z),x)$$

Part 1 : Inverting the Generator

Combination of Losses

We calculate the perceptual loss by measuring the L2 distance of the conv_5 features. During optimization, we adjust all latents over 6000 iterations using the L-BFGS method.

When relying solely on L1 loss, the resulting images tend to be blurry because this loss function averages across all pixels and is less effective at penalizing local discrepancies. Conversely, using only perceptual loss yields slightly inferior details, particularly in the upper right corner of images, as the employed convolutional features may not fully capture every nuance.

First we begin with StyleGAN and vary the perceptual loss weight from 0 to 10 while keeping the L1 loss constant at 10:

Second we fix the perceptual loss at 0.01 and vary the level of L2 regularization. L2 regularization is implemented as the averge L2 norm of delta over the batch.

From the above results it can be seen that tto low a regularization weight results in minimal change from the original image.

Comparison of Different GANs

From the above results, it can be seen that the StyleGAN performs better

Comparison of Different Latent Spaces

In StyleGAN, optimizing the z latent space presents challenges because gradients must propagate through an additional mapping network. This often leads to reconstructed images closely resembling the initial image rather than the desired input. However, switching to w or w+ latent spaces significantly improves outcomes. Among these, w+ stands out due to its expressiveness and optimization flexibility. While mean initialization benefits z space latents by reducing the expected distance between the mean and optimal latents, for w and w+ spaces, both mean and random initializations yield similar results, as optimization is comparatively easier

Part 2 : Sketch to Image

Results

Discussion

The results indicate that denser sketches, with more or larger color blocks, tend to yield generated images with reduced details in those color blocks. This effect is influenced by the L1 loss, which increases due to the additional color blocks but does not enhance perceptual information. Consequently, the generated images resemble the sketch more closely but lack fine details.

On the other hand, sparse sketches can subtly alter the layout of objects in the generated images. The choice of sketch density is also tied to hyperparameters. For instance, varying the perceptual loss value significantly impacts the quality of generated images, even when using the same mask

Part 3 : Stable Diffusion

In our task, we follow an approach similar to SDEdit. Instead of starting with random sampling and iteratively denoising using a pre-trained diffusion model, we convert the input image into noise through forward diffusion. Then, we use the DDPM sampler to denoise the noisy input, reconstructing a realistic image while staying true to the initial input guidance

Results

Comparison of Different Amounts of Noise Added

The below sketch and prompt was fixed

The amount of noise added (number of timesteps of noise addition) was varied below:

The above results show that for small amounts of noise addition, the generated image resembles the original sketch image. This is because small amounts of noise still retain the original sketch information. However, as the amount of noise increases, the generated image becomes more abstract and less like the original sketch.

Comparison of Different Guidance Strengths

The same sketch and prompt was fixed as the previous section

The amount of guidance strength to direct the generation towards the text prompt was varied below:

---