Assignment #3 - Cats Generator Playground

Background

This assignment includes two parts: in the first part, we implement a specific type of GAN designed to process images, called a Deep Convolutional GAN (DCGAN). We train the DCGAN to generate grumpy cats from samples of random noise. In the second part, we implement a more complex GAN architecture called CycleGAN for the task of image-to-image translation (described in more detail in Part 2). We train the CycleGAN to convert between different types of two kinds of cats (Grumpy and Russian Blue), and between apples and oranges.

Part 1: Deep Convolutional GAN

For the first part of this assignment, I implement a slightly modified version of Deep Convolutional GAN (DCGAN). A DCGAN is simply a GAN that uses a convolutional neural network as the discriminator, and a network composed of transposed convolutions as the generator. In the assignment, instead of using transposed convolutions, we use a combination of a upsampling layer and a convoluation layer to replace transposed convolutions.

Question:
In each of the discriminator convolutional layers, we downsample the spatial dimension of the input volume by a factor of 2. Given that we use kernel size K = 4 and stride S = 2, what should the padding be? Write your answer on your website, and show your work (e.g., the formula you used to derive the padding).
Answer:
We know that the formula for the output image size for a convolution (from the PyTorch website) looks like this: $${\text{image\_size}}_{out}=\lfloor \frac{{\text{image\_size}}_{in}+2\times \text{padding}-\text{dilation}\times (\text{kernel\_size}-1)-1}{\text{stride}}+1\rfloor$$ Substituting in all our variables, since we are downsampling the spatial dimension of the input volume by a factor of 2, should we get: $$\frac{{\text{image\_size}}_{in}}{2}=\lfloor \frac{{\text{image\_size}}_{in}+2\times \text{padding}-(4-1)-1}{2}+1\rfloor =\lfloor \frac{{\text{image\_size}}_{in}}{2}+\text{padding}-1\rfloor$$ , which implies that padding = 1.

Data Augmentations

For the deluxe version of data augmentation, I follow the standard data augmentations and just resize the image by scaling its dimensions by a factor of 1.1, then doing a random crop to the input image size, before applying a random horizontal flip.

Differentiable Augmentations

I apply differentiable augmentation with the policy of 'color,translation,cutout' on both real and fake images, and obtain the results below.

Here are the results after running the DCGAN on cat/grumpifyBprocessed with --data_preprocess=basic, with and without applying differentiable augmentations:

Without DiffAug

With DiffAug

Here are the results after running the DCGAN on cat/grumpifyBprocessed with --data_preprocess=deluxe, with and without applying differentiable augmentations:

Without DiffAug

With DiffAug

Notice that the output when the model is trained with differentiable augmentations is much better than when it is trained without - the generated images are much sharper, details such as the eyes and the fur texture are clearly visible, and the quality/variety of images is improved in general.

In terms of implementation, the two augmentation schemes (data vs differentiable augmentation) differ in where they are applied. In our implementation of DCGAN, the normal data augmentation is only applied on the real images, while differential augmentation is applied to both the real images and the fake images that were generated by the generator (G(z)). Not only that, another difference is that differentiable augmentation is, as the name suggests, composed of differentiable operations, which means we can do auto-diff and back-propagate gradients through the augmentation, thus allowing us to apply it on G(z) without breaking the back-propagation, a feature which normal data augmentation does not offer.

Results

To produce the loss curves below, we apply tensorboard's 0.99 smoothing.

Here are the loss curves with --data_preprocess=basic and without differential augmentation:

Discriminator

Generator

Here are the loss curves with --data_preprocess=basic and with differential augmentation:

Discriminator

Generator

Here are the loss curves with --data_preprocess=deluxe and without differential augmentation:

Discriminator

Generator

Here are the loss curves with --data_preprocess=deluxe and with differential augmentation:

Discriminator

Generator

For the GAN to train well, there needs to be a balance between the discriminator's ability to tell fake images from real and the generator's ability to generate realistic images, and both need to get better over time. Thus, the curves should have both discriminator and generator loss generally decreasing over time. Less-than-ideal scenarios would have the generator loss increasing instead of plateau-ing (meaning that the GAN-generated images are not realistic enough to fool the discriminator). The differential augmentation curves above are close to ideal in that the generator loss is slowly decreasing over time while the discriminator loss stays roughly stagnant.

Here are samples from early in training compared with later on in training with --data_preprocess=deluxe and differentiable augmentation enabled:

Iteration 200

Iteration 1000

It is clear to see that iteration 200 looks nothing like a cat, and the model is just starting to learn the low-level features such as the white and brown/black blobs. The blobs are quite diverse in terms of location, but the quality of the locations (even if you squint most of the images don't seem to form cats) do not seem very high. Contrast this to iteration 1000, where we can clearly see cat-like (higher-level) features, such as the nose, eyes, nose-bridge-fur-texture (the white triangle in the middle of the fact), and ears. The locations of these features also seem more structured and accurate/cat-like than the earlier iteration, and in general the quality of the image has increased quite a bit, with better (edge) contrast and more defined cat-ness.

Part 2: CycleGAN

I implement the CycleGAN architecture, with a generator that translates from image to image, and a patch-based discriminator. In all experiments below, I use the deluxe data augmentation with differential augmentation.

1,000 Iterations

Here are the results (at iteration 1,000) after training the CycleGAN for 1,000 iterations without cycle consistency loss:

X-Y

Y-X

Here are the results (at iteration 1,000) after training the CycleGAN for 1,000 iterations with cycle consistency loss:

X-Y

Y-X

10,000 Iterations

Here are the X-Y results for the grumpifyCat dataset after training the CycleGAN for 10,000 iterations:

Without Cycle Consistency

With Cycle Consistency

Here are the Y-X results for the grumpifyCat dataset after training the CycleGAN for 10,000 iterations:

Without Cycle Consistency

With Cycle Consistency

Here are the X-Y results for the apple2orange dataset after training the CycleGAN for 10,000 iterations:

Without Cycle Consistency

With Cycle Consistency

Here are the Y-X results for the apple2orange dataset after training the CycleGAN for 10,000 iterations:

Without Cycle Consistency

With Cycle Consistency

Cycle Consistency Loss

From the above images, we see that cycle consistency loss in general leads to better results.

For example, in the grumpifyCat case, a close look at the differences between the cycle consistency version and the non-cycle consistency version will reveal that the cycle consistency version manages to better preserve important image content and structure such as the pose/orientation of the cats' head, face, and body. It also leads to slightly improved image quality, where the cycle consistency cats are more well-defined and have better (edge) contrast.

In the case of apple2orange, we see that with the cycle consistency loss, the structure of images is retained much better, and less artifacts are present in the generated image. For example, look at the Y-X image with the girl in it. Without cycle consistency, the girl gets distorted in color and shape, while the oranges on the table aren't exactly transformed into apples, but with cycle consistency, we have a still-recognizable girl, with nice looking apples on the table. In the X-Y images, we see that the structure/texture of oranges generated with the cycle consistency loss are much better aligned to the apple image and also not blurry/jittery like the non-cycle consistency version.

One reason why cycle consistency leads to better results is that it acts as a form of regularization. It imposes constraints on the generators by requiring reconstructed images to be close to the original input images. This regularization helps prevent overfitting and encourages the model to learn more generalizable and invertible mappings between the two classes, which helps in generating more realistic and coherent translations. The constraints also ensure that important features like the shape, pose, and texture of a class are retained when generating images in the other class, which leads to high quality, more semantically meaningful images.

DC vs Patch Discriminator

Here are the X-Y results for the grumpifyCat dataset after training the CycleGAN for 10,000 iterations:

DC Discriminator

Patch Discriminator

Here are the Y-X results for the grumpifyCat dataset after training the CycleGAN for 10,000 iterations:

DC Discriminator

Patch Discriminator

Here are the X-Y results for the apple2orange dataset after training the CycleGAN for 10,000 iterations:

DC Discriminator

Patch Discriminator

Here are the Y-X results for the apple2orange dataset after training the CycleGAN for 10,000 iterations:

DC Discriminator

Patch Discriminator

The above images were all generated with cycle consistency loss used.

The DC discriminator generates images that are of a lower quality than that of the patch discriminator. It seems to tint the whole image in a single color, and this is most evident in the Y-X results for grumpifyCat and the X-Y results for apple2orange, where the image is quite homogeneously tinted grey and orange respectively. The shape, structure, and texture of images generated is also worse for the DC discriminator, with the images looking pixellated, distorted, or blurred (e.g., some of the cats don't really look like cats anymore, or look like severely deformed cats). On the other hand, the patch discriminator generates images that are crisp, with better definition, and a wider variety of colors that are more realistic.

The reason that the patch discriminator performs better is because it classifies smaller image patches rather than the whole image. By looking at local patches, the patch discriminator can capture fine-grained details. This allows for better discrimination between real and fake images because it focuses on smaller patches rather than the entire image at once, which in turn allows for better training of the generator due to better patch-based feedback (loss), since the feedback will help the generator better generate realistic details and textures. Contrast this to the DC discriminator, which looks at the image as a whole, leading to possible difficulties in capturing subtle differences or local inconsistencies in the image, and thus complex images where the global context can overshadow smaller local details may pose a challenge, leading to artifacts such as the tinting and blurring/distortion of textures.

Bells and Whistles

Diffusion Model

I implement the diffusion model using the skeleton provided, with a standard UNet architecture. I used the deluxe augmentation but did not apply differential augmentation. Here is a sample batch of outputs from the diffusion model after training for 2000 epochs:

The image below shows the diffusion process for the first 8 images in the batch above, where we show the output at every 50 timesteps in the diffusion process.

Extra Dataset/GIF

I run the vanilla GAN on the Monet folder in the monet2photo dataset obtained from this repo, using conv_dim=64, and obtained the following nice results (I took the samples generated in the last few epochs and animated them into a gif, since it kind of reminded me of the four seasons):