16726 Assignment 5

by Zi Wang (ziwang2)

Overview

In this project, I apply various image manipulation techniques on a natural image manifold. Firstly, I use an existing pre-trained generator to reverse engineer and identify a latent variable that accurately recreates a specified real image. Secondly, I transform a hand-drawn sketch into a corresponding realistic image. Thirdly, I use a pre-trained Stable Diffusion model to transform the given input image into noise through the forward diffusion process and then iteratively denoise to generate a realistic image using a pre-trained diffusion model.

Bells & Whistles (Extra Points)

1. I interpolated between two latent codes in the GAN model, and generate image sequences (2pts)
2. I experimented with high-res models of Grumpy Cat (2 pts)

Part 1: Inverting the Generator [30 pts]

In this section, I tackle the challenge of reconstructing an image from a specific latent code by solving an optimization problem. I utilize the LBFGS solver, which is adept at handling non-convex optimization issues.

Ablation on Loss Weights

The loss function L(G(z), x) is defined as:

$$ L(G(z), x) = w_{\text{perc}} L_{\text{perc}} + w_{L1} L_1 $$

where:

• $ w_{\text{perc}} $ and $ w_{L1} $ are weights,
• $ L_{\text{perc}} $ is the perceptual loss that I used in assignment 4,
• $ L_1 $ is the L1 loss.

Perceptual loss evaluates the content differences between two images by analyzing feature distances at specific network layers. In this project, the conv_5 layer of a pre-trained VGG-19 network is utilized to assess these distances between the input and target images. I also conduct a series of ablation experiments exploring different combinations of perceptual and L1 losses, fixing the L1 loss weight at 10 and varying the perceptual loss weight. I use StyleGAN and the latent space z yhroughout these experiments.

Target	0.0001	0.001	0.01	0.1	1	10

Finally, I choose perc_wgt=0.01 in the following experiments since it achieves the best performance.

Ablation on Generative Models

I evaluate two distinct generative models: a basic GAN and StyleGAN. Using the same optimization cycles (1000) and identical latent spaces (z), StyleGAN outperforms the vanilla GAN in reconstruction quality. This superior performance stems largely from StyleGAN's specialized architecture, which is designed for style transfer and incorporates adaptive instance normalization.

Target	Vanilla GAN	StyleGAN

Ablation on Latent Space

In my experiment with StyleGAN, I analyzed various latent spaces: z, w, and w+. While w+ and w spaces produced similar outcomes, w+ was slightly smoother and excelled in reconstructing shapes and colors. This advantage in w+ likely stems from its ability to integrate different w vectors tailored for each layer of StyleGAN, enhanced by AdaIn's layer-specific adjustments.

Target	z	w	w+

Based on the above experiments, using StyleGAN and latent space w+ achieves the best performance. Using a single 4090 GPU, the vanilla GAN around takes 6 seconds to run, while styleGAN's runtime is around 20 seconds.

Part 2: Scribble to Image [40 Points]

In this section, we integrate color scribbles as constraints in the image generation process. Specifically, these scribbles serve as a mask, shaping the L1 loss into the formula $∥M∗G(z)−M∗S∥_1$, where M represents the mask and S denotes the scribble. Here I demonstrate the outputs of using scribble constraints in image generation with StyleGAN.

Scribble	Mask	StyleGAN w+

When using StyleGAN with sparse sketches, the model cannot generate realistic outputs, as the minimal silhouettes and edges in the sketches do not provide sufficient constraints for optimizing the latent code. Conversely, with denser sketches, StyleGAN achieves more plausible results with latent space w+, which benefits from varying w vectors across different layers, aiding in a more accurate reconstruction of the input image.

Part 3: Stable Diffusion [30pts]

In this part, I implmented the approach similar to SDEdit, in which the image input serves as a “guide,” transforming the given input image into noise through the forward diffusion process instead of starting with random sampling and then iteratively denoising to generate a realistic image using a pre-trained diffusion model. In addition, I used the DDPM sampling method with the Classifier-free Diffusion Guidance. It's easy to follow the fomulas in the assignment website to reproduce the algorithm. Please refer to my code for details.

Here are some visualizations:

Prompt: "Grumpy cat reimagined as a royal painting"

Input	Output

When strength=15, I tried different different amounts of noises added to the input.

Input	timesteps=500	timesteps=700	timesteps=1000

When timesteps=1000, I experimented with different classifier-free guidance strength values.

Input	strength=5	strength=10	strength=15	strength=20	strength=25

Bells & Whistles

Interpolate between two latent codes in the GAN model, and generate an image sequence (2pt)

In this section, I perform image interpolation by first converting two images into their respective latent vectors. These vectors are then combined linearly using the formula: $ z' = \theta z_1 + (1-\theta) z_2 $, where $ \theta $ varies from 0 to 1, and $ z_1 = G^{-1}(x_1) $, $ z_2 = G^{-1}(x_2) $ represent the latent vectors derived from the inverse transformations of images $ x_1 $ and $ x_2 $.

Experiment with high-res models of Grumpy Cat (2 pts)

I carried out experiments on generating images of grumpy cats at a resolution of 256x256, and the results are detailed below.

Target	Output