Assignment #3 - When Cats meet GANs

Description

In this project, we will explore the Generative Adversarial Networks (GANs) for image generation. In the first part, we will implement a modified version of Deep Convolutional GAN (DCGAN), where we use a convolutional neural network as the discriminator and generator, and measure it's performance on the grumpy cat dataset. Meanwhile, we will also explore different data preprocessing strategies as well as differential data augmentation techniques for better performance. In the second part, we will implement CycleGAN for unsupervised image-to-image translation. We will compare the performance using patch-discriminator with DC-discrimiator, and will also report the performance on apple2orange and orange2apple datasets.

For Bells & Whistles, we will first show some generation results from existing diffusion models. Next we will experiment with several state-of-the-art image editing methods to compare their performance on domain translation with CycleGAN.

Part 1: DCGAN

Implementation

1. Data Augmentation

DCGAN will suffer from overfitting without proper data augmentation on a small dataset. Therefore, we will be implementing some data augmentation, including random crop and random horizontal flip as the deluxe version of data augmentation.

2. Model Implementation

Discriminator of the DCGAN

The structure of the discriminator is shown in the figure. We need to calculate the padding size: \( (N + 2 * P - K)/S + 1 = N/2 \), where \(K=4, S=2\) Then we can get \( P=1 \). Therefore, we should implement the Convolutional layer with kernel_size=4, stride=2, padding=1.

Generator of the DCGAN

The structure of the generator is shown in the figure. For the first layer, it's better to directly apply convolution to get the \( 4 \times 4 \) output. Therefore, I am using nn.ConvTranspose2d with kernel_size=4, stride=1, padding=0 as the first layer. For the remaining layers, I used the up_conv function defined in the codebase with kernel_size=3, stride=1, padding=1. I also passed in the correct normalization function and activation function as described in the figure.

Results

1. Training Loss

DCGAN-basic

DCGAN-basic-diffaug

DCGAN-deluxe

DCGAN-deluxe-diffaug

For GAN training, the generator should learn to generate samples that are more and more similar to the real samples, so the loss should go down. The discriminator should be getting better at distinguishing between real and fake images, so the loss should also go down. When converged, the two losses should be relatively stable.

2. Training Results

With --data_preprocess=deluxe and differentiable augmentation enabled, here are one sample from early in training (iter=200) and one from later in training (iter=6400).

Sample at 200

Sample at 6400

During early stages, the generated images are blurry and look very different from the real grumpy cats. As training progresses, the generated images are becoming clearer, of better quality, and are getting more and more similar to the real samples.

Part 2: CycleGAN

Implementation

1. Data Augmentation

For data augmentation, we are using the deluxe data preprocessing method as used for DCGAN training.

2. Discriminator

We implement the patch discriminator for CycleGAN, which performs classification on patches of an image instead of the whole image. This will allow the model to focus on local structure better.

3. Generator

For generator, we are implementing the network as described in the following figure: cycle-generator

4. Cycle-consistency Loss

One key contribution of the CycleGAN paper is to propose the cycel-consistency loss, which guides the model to translate from domain X to Y and then translate back from Y to X', where X' should be close to X.

Cycle-Consistency Loss Results

1. Result for 1000 iterations on Grumpy Cats

Training Without Cycle-Consistency Loss

X-Y (1000 iterations)

Y-X (1000 iterations)

Training With Cycle-Consistency Loss

X-Y (1000 iterations)

Y-X (1000 iterations)

2. Result for 10000 iterations on Grumpy Cats

Training Without Cycle-Consistency Loss

X-Y (10000 iterations)

Y-X (10000 iterations)

Training With Cycle-Consistency Loss

X-Y (10000 iterations)

Y-X (10000 iterations)

3. Result for 10000 iterations on Apple2Orange

Training Without Cycle-Consistency Loss

X-Y (10000 iterations)

Y-X (10000 iterations)

Training With Cycle-Consistency Loss

X-Y (10000 iterations)

Y-X (10000 iterations)

Without the cycle-consistency loss, the images are of the correct color but are usually very blurry and don't contain much details. This is because without the cycle-consistency loss, there wouldn't be a one-to-one mapping between two domains, therefore the image translation are not very accurate.

DC-Discriminator and patch-Discriminator Results

1. Result for 10000 iterations on Grumpy Cats

Training with DC-Discriminator

X-Y (10000 iterations)

Y-X (10000 iterations)

Training with Patch-Discriminator

X-Y (10000 iterations)

Y-X (10000 iterations)

2. Result for 10000 iterations on Apple2Orange

Training with DC-Discriminator

X-Y (10000 iterations)

Y-X (10000 iterations)

Training with Patch-Discriminator

X-Y (10000 iterations)

Y-X (10000 iterations)

From the results, we can see that a Patch-Discriminator can provide more fine-grained feedback to the generator, which leads to better generated images. In contrast, the DC Discriminator requires less computation and memory.

Bells and Whistles

For bells and whistles, I first experimented with existing pre-trained diffusion models for text-conditioned image generation. Next, I also experimented with several state-of-the-art image editing algorithms and report their performance on domain translation.

Generating Images from Pre-trained Diffusion Model

I am using stable-diffusion-v2.1 to generate the following two images. The images generated by Stable-Diffusion are of very high quality and contain much details.

"A photo of a grumpy cat"

"A photo of a russian blue cat"

State-of-the-art Image Editing Results

Next, I experimented a state-of-the-art image editing method: InstructPix2Pix.

InstructPix2Pix Results

The first row is the original image, the second row is the edited image. I am using the prompt: "turn the A into a B", where A and B are from (apple, orange, grumpy cat, russian blue cat).

We can observe that for InstructPix2pix, which is a model trained on paired image datasets, the structure of the original image is well preserved. However, because there is no such idea as "domain" during InstructPix2Pix's training, the edited images will have very diverse appearances that align well with the editing text prompt.