**Cats Generator Playground** Introduction =============================================================================== In this assignment, we implement GANs to generate images of cats. We first use a Deep Convolution GAN (DC-GAN) to generate images of grumpy cats. We experiment with different augmentation techniques. Next we use a cycleGAN to "translate" between two types of cats. We also show some images of grumpy cats using a pre-trained diffusion model. DC-GAN =============================================================================== Discriminator ------------------------------------------------------------------------------- The DC-GAN has a generator and a discriminator which play a "zero-sum game" during training. The discriminator has the following architecture

We want each of the layers of the discriminator to halve the spatial resolution. So we need to set an appropriate kernel size and padding of the convolution layers. We can write the relation between the output and input spatial dimension as $O = \cfrac{I - K + 2P}S + 1$, where where O is the output size, I is the input size, K is the kernel size, P is the padding, and S is the stride. Setting $O = I/2$ and $S = 2$, we get $I/2 = I/2 - K/2 + P + 1$
or $P = K/2 - 1$
Since we use kernels of size 4 in our model we will use padding of size 1 in all the layers except the last one where are reducing the dimension by a quarter. Plugging $O = 1$, $I = 4$, $K = 4$ and $S = 2$, we get $P = 0$. So we use zero padding in the last layer. | Layer | Input Size | Output Size | Kernel Size | Stride | Padding | | ----- | ---------- | ----------- | ----------- | ------ | ------- | | conv1 | 3×64×64 | 32×32×32 | 4×4 | 2 | 1 | | conv2 | 32×32×32 | 64×16×16 | 4×4 | 2 | 1 | | conv3 | 64×16×16 | 128×8×8 | 4×4 | 2 | 1 | | conv4 | 128×8×8 | 256×4×4 | 4×4 | 2 | 1 | | conv5 | 256×4×4 | 1×1×1 | 4×4 | 1 | 0 | Generator ------------------------------------------------------------------------------- The generator has the following architecture

In the first layer, instead of using the upsample layer, a convolution layer with kernel size 4, stride 1 and padding size 3 was used. This will give us a 4x4 ouput from the 1x1 input. Training with data augmentation ------------------------------------------------------------------------------- Training loss comparison -------------------------------------------------------------------------------

A few notes on the loss curves: 1. Differentiable augmentation clearly performs better since the generator is able to achieve a lower loss, and conversely, the discriminator has a harder time differentiating between reals and fakes. 2. With deluxe preprocessing, on average the performance is slightly better (with a slightly lower loss for the generator, and a slightly higher loss for the discriminator respectively). Generated Results ------------------------------------------------------------------------------- The results from the DC-GAN were generated using two different sets of augmentations: 1. The basic augmentation had just resizing and normalization 2. The deluxe augmentation has random-crop, color-jitter and random horizontal flip in addition to the basic augmentations. The results with these augmentations are shown below.

We can see the results generated with deluxe augmentation have better quality. Next we add differentiable augmentation (diffAugment) in addition to the basic and deluxe augmentations. The results are shown below.

We can also compare the images with diffAugment with basic and deluxe pre-processing and see that the quality is pretty similar. It is evident that diffAugment is responsible for the noticeable improvement across the board. Now we show how the generated results change as the training progresses while training with deluxe pre-processing and diffAugment. We can see that as the training progresses the GAN focusses on generating more lower level details. In the initial iterations, it just creates a structure resembling a cat, as in the later iterations it focusses on creating the ears and the eyes.

Cycle GAN =============================================================================== A CycleGAN enables the translation of an image from one domain to another without requiring paired training data. Like in a traditional GAN, a CycleGAN consists of two generator networks (G and F) and two discriminator networks (D_X and D_Y). The generator G maps images from domain X to domain Y, while F maps images from domain Y to domain X. In addition to the adversarial loss used in traditional GANs, CycleGAN introduces a cycle-consistency loss. This loss ensures that when an image is translated from one domain to another and then back again, the resulting image should be close to the original image. This is achieved by minimizing the reconstruction error between the original image and the cycle-reconstructed image. The cycle-consistency loss is computed for both forward and backward cycles. The forward cycle takes an image from domain X, translates it to domain Y using G, and then translates it back to domain X using F. The backward cycle does the opposite, starting with an image from domain Y. The full objective function of a CycleGAN combines the adversarial losses from the two discriminators (D_X and D_Y) and the cycle-consistency losses from the forward and backward cycles. The generators (G and F) are trained to minimize this objective, while the discriminators are trained to maximize it. Early Results ------------------------------------------------------------------------------- Here are some of the earlier results of our CycleGAN with patch discriminator on the first dataset of cats (Grumpy and Russian Blue). These are results after a 1000 iterations, and we have two sets of results without and with cycle consistency loss.

We can see a marked improvement in results using the cycle-consistency loss, so we train it for more iterations. Extended Results ------------------------------------------------------------------------------- We first compare the results on two datasets with and without cycle-consistency loss Shown below are the results for 10000 iterations on the Cats dataset

Shown below are the results for 10000 iterations on the Apples2Oranges dataset

We can see from the above examples that using cycle-consistency loss leads to far superior results. The difference is especially noticeable in the eye region of the cats, and in the shape of the oranges and apples. The cycle-consistency loss preserves the original shape of the apples and oranges better while changing their color. Second, we compare the generated results of the cycle GAN using two different discriminators - the original patchGAN discriminator and the DC-GAN discriminator. Results on the Cat dataset

Results on the Apples2Oranges dataset

The PatchGAN discriminator gives superior results when used in the CycleGAN as it pays attention to both local (patche-based) and global features, while the DC-GAN dicriminator only cares about the global features. We can see in case of the DC-GAN discriminator, the color bleeds into the background more often. Bells and Whistles =============================================================================== Generated samples from a pre-trained diffusion model ------------------------------------------------------------------------------- Here are some grumpy cats generated using Stable-Diffusion-v1-4

Spectral Norm for stable training ------------------------------------------------------------------------------- We use spectral norm in the discriminator of the DC-GAN to see if it leads to more stable training. We compare the loss curves of the DC-GAN training with and without spectral norm below. The green plot is for training with spectral norm and the orange plot is without it

We can see that spectral norm does indeed lead to more stable training, especially in case of the generator. We compare the results from the two trainings below after 6400 iterations.