In this assignment, I implemented image editing methods with 1) GANs and 2) Stable Diffusion. First, I implemented GAN inversion with a vanilla GAN and a StyleGAN model to reconstruct images from their latent codes. I then implemented scribble to image editing by optimizing the latent code of a GAN with a color scribble constraint. Next, I implemented image editing with the Stable Diffusion model by adding noise to an input image using the forward diffusion process and then denoising the noisy image with a text prompt condition.
I also implemented one of the Bells and Whistles: Interpolating between 2 latent codes in the GAN model (included at the bottom of this page).
I inverted the GAN generator by solving an optimizaiton problem to reconstruct the image from a given latent
code. I used
the LBFGS optimizer to minimize the loss between the reconstructed image and the original image.
The loss function is a combination of perceptual loss and Lp loss:
\[ L(G(z),x)=w_{perc} L_{perc} + w_{Lp} L_{p} \]
where \(L_{perc}\) is the perceptual loss, \(L_{p}\) is the Lp loss, $w_{perc}$ is the weight for the perceptual
loss, and $w_{Lp}$ is the weight for the Lp loss.
I used the conv_5 layer of the VGG19 model to compute the perceptual loss.
I compared L1 and L2 losses for the reconstruction of the image from the latent code. I kept the perceptual loss weight at 0.01, the Lp loss weight at 10, and the model as StyleGAN w+ for both settings. The results are shown below.
We can see that the L1 loss provides a better reconstruction of the image compared to the L2 loss. This may be because the L1 loss promotes sharper edges in the reconstructed image while the L2 loss may blur the features.
I compared the reconstruction of the image from the latent code using the vanilla GAN and StyleGAN. I kept the latent space in z, perceptual loss weight at 0.01, the Lp loss weight at 10, and the L1 loss for both settings. The results are shown below.
We can see that StyleGAN provides a better reconstruction of the image compared to the vanilla GAN. This may be because the StyleGAN architecture is designed for style transfer and uses adaptive instance normalization.
I compared reconstruction results for different latent space choices: z, w, and w+. I kept the model as StyleGAN, perceptual loss weight at 0.01, the Lp loss weight at 10, and the L1 loss for all settings. The results are shown below.
We can see that the w+ latent space provides a better reconstruction of the image compared to the z and w latent spaces. This is probably because the w+ latent space provides more control over the style of the image at each layer of StyleGAN.
I implemented scribble to image editing by optimizing the latent code of a GAN with a color scribble constraint. I also used the LBFGS optimizer, but I added the scribble constraint so that within a mask, corresponding pixels in the scribble and the generated image have the same color. I used the best image reconstruction configuration from the previous experiments: StyleGAN model, w+ latent space, L1 loss, perceptual loss weight of 0.01, and L1 loss weight of 10.
Below are some scribble to image results:
Denser sketches have more pixels that are constrained by the scribble, so the generated image sometimes cannot match the natural image manifold and may look less realistic compared to generated images from sparser scribbles. Also, sketches that use larger areas of blue and pink lead to less realistic generated images because real cat images do not have such large areas of blue and pink.
To implement image editing with the Stable Diffusion model, I added noise to an input image using the forward diffusion process and then denoised the noisy image with a text prompt condition. I implemented this using DDPM sampling and classifier-free guidance.
Below are guided image synthesis results on 2 different numbers of denoising timesteps (the more timesteps, the more noise added) and 2 different classifier-free guidance strength values.
We can see that as the classifier-free guidance strength increases, the generated image becomes more aligned with the text prompt. Also, as the number of denoising timesteps increases, the generated image is allowed to become more different from the input image and more aligned with the text prompt.
I implemented interpolating between 2 latent codes in the GAN model. I used the best StyleGAN w+ model configuration and interpolated between 2 latent codes by first optimizing for the latent representations, then linearly interpolating between the latent codes, and finally generating images from the interpolated latent codes. Two image interpolations are shown below.
