Assignment 3

Overview

Part 1: Implementation of Deep Convolutional GAN (DCGAN) to generate grumpy cats from samples of random noise. DCGAN uses a convolutional neural network as the discriminator and a network composed of transposed convolutions as the generator.
Part 2: The second part consists of training Cycle-GAN for the task of image-to-image translation (apples to oranges). The generator in Cycle-GAN first encodes the input via a series of convolutional layers and then transforms the features by passing them through a series of residual blocks. The transformed features are then decoded using transposed convolutional layers.

Part 1: Deep Convolutional GAN

Given this generator-discriminator architecture, the following objective function is used for training \[ \mathcal{L}_{\text{GAN}}(G, D) = \mathbb{E}_{z \sim p_z(z)}[\log D(G(z))] \] \[ \mathcal{L}_{\text{D}}(D, x, z) = -\mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] - \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \]

Padding

For the convolutional layers of the discriminator, the padding is calculated as follows
\( K = \text{kernel_size} \quad P = \text{padding} \quad S = \text{stride} \)

\[ \text{Output size} = \left \lfloor \frac{\text{Input size} - K + 2P}{S} + 1 \right \rfloor \] \[ 32 = \left \lfloor \frac{64 - 4 + 2P}{2} + 1 \right \rfloor \] \[ 32 = \left \lfloor 30 + P + 1 \right \rfloor \] \[ P = 1 \]

Training Loop

Loss Curves

Loss curves when the GAN manages to train

As seen from the loss curves, the generator/discriminator loss never converges to zero if the GAN is able to train successfully. This is because the generator and the discriminator are in a constant state of competition. When the generator is able to generate realistic images, the discriminator is also able to distinguish between real and fake images with high accuracy. GANs are often difficult to train due the phenomenon of mode collapse where the generator learns to generate only a few samples and the discriminator is unable to distinguish between real and fake samples. This is often observed in the early stages of training. In such cases the discriminator loss goes down to zero and the generator loss converges to one.

Also, the difference in training with and without differentiable augmentation can be clearly seen from the loss curves. The generator loss goes down significantly allowing the model to train faster and generate better samples.

Generated Samples

Quality of samples and how they improve through training

Aesthetically, the generated samples improve a lot as the training progresses. In particular, high frequency features start to appear rather than just low frequency blobs. The blurriness in the samples is reduced and by the end of the training loop, sharp images are generated. Also, as apparent from the loss curves, using differential augmentation produces significantly better images.

--data_preprocess deluxe

--data_preprocess deluxe --use_diffaug

Difference between the augmentation schemes

The key idea behind differential augmentation as suggested in the paper is that to propagate gradients through the augmented samples to G, the augmentation/transforms (T) must be differentiable. In vanilla augmentation schemes, only the real images are augmented. This causes the model to learn a different data distribution than intended. Using DiffAugment both real and fake samples are augmented and this effectively prevents the discriminator from over-fitting.

Part 2: Cycle-GAN

CycleGAN is a type of generative adversarial network (GAN) designed for unsupervised image-to-image translation tasks. It aims to learn mappings between two different domains without paired training data. Instead of directly mapping images from X to Y and vice versa. CycleGAN employs a cycle-consistency loss to enforce the translated images to be consistent when translated back to their original domains.
It does so by optimizing the following objective functions: \[ \mathcal{L}_{\text{total}}(G, F, D_X, D_Y, X, Y) = \mathcal{L}_{\text{GAN}}(G, D_Y, X, Y) + \mathcal{L}_{\text{GAN}}(F, D_X, Y, X) + \lambda \mathcal{L}_{\text{cycle}}(G, F) \] \[ \mathcal{L}_{\text{GAN}}(G, D_Y, X, Y) = \mathbb{E}_{y \sim p_{\text{data}}(y)}[\log D_Y(y)] + \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log (1 - D_Y(G(x)))] \] \[ \mathcal{L}_{\text{cycle}}(G, F) = \mathbb{E}_{x \sim p_{\text{data}}(x)}[||F(G(x)) - x||_2] + \mathbb{E}_{y \sim p_{\text{data}}(y)}[||G(F(y)) - y||_2] \]

Experiments

--train_iters 1000

--train_iters 1000 --use_cycle_consistency

Full training : Grumpy cats

--train_iters 10000

--train_iters 10000 --use_cycle_consistency

Full training : Fruits

--train_iters 10000

--train_iters 10000 --use_cycle_consistency

Difference in results with and without cycle consistency. Explain why there is or isn't any difference.

Positives: With cycle consistency loss, more features of the input domain are preserved in the translated domain. This is clearly noticeable in the results of grumpy cats.
Negatives: Cycle consistency loss can sometimes lead to overly rigid mappings between the input and output domains, resulting in loss of diversity in the generated samples. It seems that using cycle consistency color scheme of the target domain is not properly learned as it might encourage the model to learn an average color between the two domains to ensure cycle consistency. This can be seen in the very first top left image of apple-to-orange generator.

Patch Discriminator v/s DCDiscriminator

All previous results for cycle gan were generated using the patch discriminator. The results below are generated using the DCDiscriminator. The training loop is the same as the previous experiments, only the discriminator is changed.

--disc dc --train_iters 10000 --use_cycle_consistency

--disc dc --train_iters 10000 --use_cycle_consistency

Observations between the results using DCDiscriminator and PatchDiscriminator

Contrary to the PatchDiscriminator, the DCDiscriminator only cares about the entire image and is bad at preserving local features. Thus the image translations are slightly worse when using the PatchDiscriminator. In the case of grumpy cats, the sometimes the colors looks unnatural and you can spot some channel artifacts. One might expect huge difference between the two discriminators but for images with such small resolution, the difference is hardly noticeable.

Bells and Whistles

Generate samples using a pretrained diffusion model

I used DreamBooth by stabilityAI to generate the following images of sports cars.