_________________________________________________________

HW5 - GAN Photo Editing

Part 1: Inverting the Generator [30 pts]

In this section, we will address an optimization challenge aimed at reconstructing an image using a specific latent code. Our experiments will explore various loss functions, different generative models, and diverse latent spaces.

The objective is to reconstruct the following image:

into

Various Combinations of Losses

The choice of loss functions is critical for effectively guiding the optimization process. In this task, we will evaluate the impact of different losses on extracting and inverting textures using a latent w+ space in StyleGAN. The ablation study will include lp losses (l1 and l2) as well as Perceptual loss, examining their effectiveness and outcomes in this context.

1000 iters with loss type l_prec = 0.01
l1 = 10
l2 = 0
l1 = 0
l2 = 10
l1 = 10
l2 = 10

We understand that the L1 loss may not be suitable for complex images with intricate details because it focuses on minimizing the absolute differences rather than preserving the overall structure. This is evident from the image reconstructed using only L1 loss, which fails to capture the purple flower behind the cat. However, L2 loss tends to produce blurry results as it smooths out sharp edges. Therefore, I will combine L1 and L2 losses for my task to balance these effects.

As for the Perceptual loss, if it is applied too strongly, the object can become blurry, making it difficult to identify the accurate latent representation. I have found that setting the Perceptual loss weight to 0.05 yields the best results.

Regarding the runtime of my code, there are no significant differences between the implementations, as all settings require calculating each loss. Each image takes approximately 26-27 seconds to generate, without much variation.

Different generative models including

Here, we test the effects of different generative models. For this scenario, I have set the weight of the Perceptual loss to 0.05 and used a combination of L1 and L2 losses for pixel loss, maintaining the same initialization of the latent space z. I will compare the results of using VanillaGAN and StyleGAN.

Our goal is to reconstruct the following image, with all results obtained after 1000 epochs.

Reconstruction Target VanillaGAN (t = 11.23 s) StyleGAN (t = 19.77 s)

It can be observed that StyleGAN produces superior results, likely due to its more disentangled latent space. However, I also noted that the runtime for a simpler model like VanillaGAN is shorter. This presents a trade-off between image quality and computational complexity.

Different latent space

I applied optimization using different latent spaces—z, w, and w+—for the StyleGAN model. I set the weight of the Perceptual loss to 0.05 and used a combination of L1 and L2 losses.

Reconstruction Target StyleGAN z (avg t = 20.01 s) StyleGAN w (avg t = 26.12 s) StyleGAN w+ (avg t = 26.31 s)

We observe that using only the latent z space, the reconstruction results are somewhat limited in details. Using w and w+ achieves similar processing times, but w+ better preserves details and color accuracy.

Which combination gives you the best result?

I would say the combination of L1 and L2 norms, with a perceptual weight of 0.05, and using StyleGAN with the latent w+ space gives the best results.

Part 2: Scribble to Image [40 Points]

I initially followed the instructions to complete Part 3, masking out the gradients outside the specified area and calculating the perceptual loss within the drawing area. The results are displayed in the third column.

However, I observed some issues with this method; primarily, the results can be very blurry. For example, when the sketch is sparse, the generated image tends to focus predominantly on the sketched areas, neglecting other parts of the cat's face, as seen in the first row of examples. Conversely, when the sketch is too dense, the image tends to overlook the finer details of the cat's face. To address this, I adjusted the perceptual loss to also consider the entire image at a certain weight. The improved results are shown in the fourth column.

Sketch target
(1000 epochs)

Stable Diffusion Model (30pts)

Show some example outputs of your guided image synthesis on at least 2 different input images.

python img2img.py --prompt "A cat with a mask" --init-img assets/stable-samples/img2img/sketch-cat-256.png --strength 0.7 --scale 15

python img2img.py --prompt "A cat with a hat" --init-img assets/stable-samples/img2img/sketch-cat-256.png --strength 0.7 --scale 15

We show a comparison of generated images using 2 different amounts of noises added to the input.

--prompt "A cat with a hat" --seed 123

--prompt "A cat with a hat" --seed 1234

Furthermore, we show a comparison of generated images using 2 different classifier-free guidance strength values.

--prompt "A cat with a hat" --strength 0.7

--prompt "A cat with a hat" --strength 0.9