Assignment 3: Cats Generator Playground

Introduction

This assignment is all about GANs. We finish the implementation of a Deep Convolutional GAN in Part 1. This GAN generates "grumpy" cats from random noise. In part two, we finish implementing a CycleGAN. Instead of generating cats from noise we will convert between different cat breeds, specifically Grumpy to Russian Blue and vice versa.

Part 1: Deep Convolutional GAN

For our Deep Convolutional GAN (DCGAN), the discriminator is a CNN. The generator is a series transposed convolutions. In our case, we will use a combination of sampling and convolutions instead of the transposed convolutions.

Implement Data Augmentation

To improve the performance of this method we augment the data set. We apply random crop and horizontal flip to augment the dataset.

Discriminator of the DCGAN

Above is the architecture of the discriminator. For the first 4 layers we use a kernel size of 4, a stride of 2 and padding is 1. For the last layer padding is 0 instead of 1.

Generator of the DCGAN

Above is the architecture of the generator. For the last 4 convolutions we use a stride of 1 and padding is 1. For the first, padding is 0 and the kernel size is 4.

Training Loop

Above is the pseudocode we implemented as the training loop.

Differentiable Augmentation

We also add in the option to use differentiable augmentation. This allows us to augment both the real and fake samples, while being able to back propagate. In theory this should allow us to better augment our dataset, resulting in a better trained model.

Experiment with DCGANs

For visualization purposes we cranked the smoothing on the graphs to 0.99.

Basic Preprocessing Without Differentiable Augmentation

Basic Preprocessing With Differentiable Augmentation

Deluxe Preprocessing Without Differentiable Augmentation

Deluxe Preprocessing With Differentiable Augmentation

When the GAN manages to train both loses should decrease. As seen in the basic example without differentiable augmentation, if the generator loss fails to go decrease the GAN will not learn properly.

With differentiable augmentation the resulting cats overall look better, in my opinion. This addition improves our ability to learn and therefore generate cats in both the deluxe and basic cases. This is because we are able to augment our data in more complex ways than cropping and flipping, this can be seen in the Real images above. Since the method is differentiable the augmentation is applied to real and fake data. This provides a better learning process than just normal preprocessing augmentation.

Deluxe Preprocessing With Differentiable Augmentation

Early in the training we are learning the general dark and light blobs that make up a "grumpy" cat. As training progresses we get more detailed. Those dark areas become detailed, the upper left and right parts of the face are more complete, with eyes and fur.

Part 2: CycleGAN

Generator

Above is the architecture of the generator. The first 2 convolutions we use a stride of 2, a kernel of size 4 and a padding of 1. For last two use a stride of 1, kernel size 3 and padding of 1.

Patch Discriminator

The patch discriminator classifies patches of images instead of whole images. To do this we remove the last convolution from the discriminator from DCGAN, and remove the normalization and activation from the fourth layer.

CycleGAN Training Loop

We implemented the above pseudocode for the training loop.

CycleGAN Experiments

CycleGAN without Cycle-Consistency (After 1000 Iters)

CycleGAN without Cycle-Consistency (After 10000 Iters)

CycleGAN with Cycle-Consistency (After 1000 Iters)

CycleGAN with Cycle-Consistency (After 10000 Iters)

When using cycle consistency (CC) we add a loss that says a real image in domain 1, converted to domain 2, then converted back to domain 1 should look like the initial real image. Both methods have their weaknesses and strengths.

CC makes the resulting image look more like original, due to loss function structure, which is usually desired when working with the cats. In the first example with CC the grey cat has the same facial structure as the grumpy cat. While without CC it just looks more like a random grey cat.

CC also allowed for more meaningful conversions between fruits, as seen in the second example. It does more than just convert the color.

Sometimes the restriction of CC is too much and causes the results to look worse than without, as seen in the third example.

Both methods fail where there is "a lot" going on in the image beyond just fruit, we can see both methods fail in the last example. CC fails more so, basically giving the whole image an orange texture. When not using CC, the model seems to do a better job of converting the fruit to the right color and ignoring the background a bit. Since CC has a loss that wants the generated image to be able to be generated back to the original, the background is often preserved in less than ideal ways that don't look natural.

CycleGAN with Cycle-Consistency & DCDiscriminator (After 10000 Iters)

We use the patch discriminator because it determines if patches of the image look real, rather than the whole, this in theory should yield more realistic results.

The results support what we expected. The images using our DCGAN discriminator are in fact of lower quality than those from the patch discriminator. They are less detailed, more "messy" looking, and in the case of the cats, they outputs look less diverse. Specific examples that show these effect well are shown below.