Assignment 3 - Cats Generator Playground

Part 1 : Deep Convolutional GAN

The Generator and Discriminator for the DC GAN is shown below:

Generator

Discriminator

Padding Calculations

Based on the above layer shapes in the discriminator, the padding is calculated to be:

Training Loop

#

DC GAN with Basic Tranforms

Discriminator

Generator

DC GAN with Deluxe Transforms

Discriminator

Generator

Analysing Loss Curves

The loss curves reveal that if a GAN trains successfully, the loss of the generator/discriminator doesn’t settle at zero. This is a result of the ongoing rivalry between the generator and the discriminator. When the generator can produce convincing images, the discriminator is equally skilled at differentiating between authentic and fabricated images. Training GANs can be challenging due to the occurrence of mode collapse, a situation where the generator starts to produce a limited variety of samples and the discriminator struggles to tell real and fake samples apart. This is typically noticeable in the initial phases of training. In such scenarios, the discriminator’s loss drops to zero while the generator’s loss approaches one.

Furthermore, the impact of training with and without differentiable augmentation is evident from the loss curves. The significant reduction in the generator’s loss accelerates the training process and enhances the quality of the generated samples.

Differentiable Augmentation Examples

Differences between Traditional Augmentation and Differentiable Augmentation

Implementation

• Traditional Data Augmentation: In traditional data augmentation, transformations such as rotation, scaling, cropping, and flipping are applied to the input images. These transformations are non-differentiable, meaning they cannot be included in the backpropagation process during training. The augmented images are used only in the forward pass, and the original images are used in the backward pass.
• Differentiable Augmentation: In differentiable augmentation, as described in the provided code, the augmentations are made differentiable so that they can be included in both the forward and backward passes during training. This is achieved by implementing the augmentations as differentiable functions in PyTorch. The augmentations include random brightness, saturation, contrast, translation, and cutout, and they are applied to both real and fake images.

Effects:

• Traditional Data Augmentation: The main purpose of traditional data augmentation is to increase the diversity of the training data and reduce overfitting. However, it does not directly contribute to the learning process because the transformations are not included in the backpropagation process.
• Differentiable Augmentation: Differentiable augmentation not only increases the diversity of the training data but also contributes directly to the learning process because the augmentations are included in the backpropagation process. This can lead to more efficient learning, especially in the context of Generative Adversarial Networks (GANs), where it can help to stabilize the training process and improve the quality of the generated images. By applying augmentations to both real and fake images, it helps to reduce overfitting in the discriminator and makes the generator more robust to the augmentations.

Part 2 : CycleGAN

Unlike traditional methods that require paired examples, CycleGAN can learn to translate between two domains using collections of images that are not directly related. It achieves this by enforcing the intuition that the learned mappings should be reversible and bijections between the source and target domains1. In essence, it allows us to transform images from one style or context to another without relying on explicit paired training data.

NOTE: I ran the CycleGAN on two different machines. Some of the outputs in comparisons are therefore different due to this reason

Generator

Discriminator

CycleGAN on 1000 Iterations

Without Cycle Consistency Loss

Using Cycle Consistency Loss

CycleGAN on 10,000 Iterations on Cats

Without Cycle Consistency Loss

Using Cycle Consistency Loss

CycleGAN 10,000 Iterations on Fruits

Without Cycle Consistency Loss

Using Cycle Consistency Loss

Differences in Results With and Without Cycle Consistency

Pros:

Cycle Consistency Loss: By enforcing cycle consistency, more features from the input domain are preserved in the translated domain. This is particularly evident in the results when transforming images of grumpy cats. The preservation of content across domains ensures that essential characteristics are retained during translation.

Cons:

Rigidity and Loss of Diversity: However, cycle consistency loss can sometimes lead to overly rigid mappings between the input and output domains. In pursuit of maintaining consistency, the model may sacrifice diversity in the generated samples. For instance, variations in color or style might be limited due to this constraint. This is visible in images like:

Color Scheme Challenges: Additionally, the model struggles with effectively learning the color scheme of the target domain. Rather than capturing the full spectrum of colors, it may converge toward an average color between the two domains. This can be observed in the image below:

Patch Discriminator vs DC Discriminator

NOTE: Cycle Consistency Loss applied on both

---

The DC Disriminator cares only about the entire image and does not prioritize local(patch) features. Due to this reason, there are some color channel artifacts as seen below:

Artifacts like above and averaged textures in DC Discriminator make it’s images look less natural.

Bells and Whistles

Generate samples using a pretrained diffusion model

I used OpenJourney which is similar to MidJourney to generate a few images shown with their prompts used for generation below:

---