**16-726
Cats Generator Playground** **Anish Jain (anishaja)** Overview =========================================================================== In this assignment, I implemeted DC-GANs to generate images of grumpy cats. I later implement Cycle-GANs to convert the grumpy cats to Russian Blue cats. I also play around with diffusion models to generate more cats. Deep convolutional GANs =========================================================================== In this segment, I tackle the implementation of the generator, discriminator, and training loop for DCGAN. Furthermore, I explore different data augmentation techniques, namely basic versus deluxe augmentation and DiffAugment. The discriminator is a CNN responsible for downsampling the input image of size 3×64×64 to a single value of 1×1×1, thereby classifying it as either fake or real. Conversely, the generator takes a random noise latent vector of size 100×1×1 and upsamples it to generate an image of size 3×64×64 using a series of Convolutional layers. Throughout both the generator and discriminator architectures, ReLU activation and Instance Normalization are employed for the Conv2d layers, with the exception of the final layer. During training, I adopt a strategy of alternating between training the generator and discriminator. This involves updating the weights of each component separately while keeping the other fixed, utilizing least-squares loss. The generator's objective is to produce images that deceive the discriminator, while the discriminator learns to better distinguish between real and fake images over time.   Padding calculation for Discriminator --------------------------------------------------------------------------- The formula for padding is $$O = \frac{W - K + 2P}{S} + 1$$ where O is the output size, W is the input size, K is the kernel size, P is the padding, and S is the stride. For the generator, the output size is half the input size, so O = W/2. For the discriminator, the output size is 1, so O = 1. Using these equations, we can solve for the padding P. $$P = \frac{(O - 1) * S + K - W} {2}$$ Using this formula, we get **P = 1 for layers conv1, conv2, conv3, and conv4** and **P = 0 for the last layer conv5**. | Layer | Input Size | Output Size | Kernel Size | Stride | Padding | |-------|------------|-------------|-------------|--------|---------| | conv1 | 3x64x64 | 32x32x32 | 4x4 | 2 | 1 | | conv2 | 32x32x32 | 64x16x16 | 4x4 | 2 | 1 | | conv3 | 64x16x16 | 128x8x8 | 4x4 | 2 | 1 | | conv4 | 128x8x8 | 256x4x4 | 4x4 | 2 | 1 | | conv5 | 256x4x4 | 1x1x1 | 4x4 | 1 | 0 | Data Augmentation --------------------------------------------------------------------------- The results indicate that utilizing deluxe augmentation yields better results compared to basic augmentation alone. Moreover, incorporating differentiable augmentation leads to a significant improvement in quality. This improvement is attributed to the fact that without differentiable augmentation, the generator struggles to replicate augmented images accurately, leading to artifacts, particularly noticeable with effects like ColorJitter. By applying DiffAugment to both real and fake images, convergence and quality are enhanced through regularization of the discriminator without altering the target distribution. ### Basic Augmentation   ### Deluxe augmentation ~~~~~~~~~~~~~~~~~~~~~~python train_transform = transforms.Compose([ transforms.Resize(osize, Image.BICUBIC), transforms.RandomCrop(opts.image_size), transforms.RandomHorizontalFlip(), transforms.RandomRotation(15), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), ]) ~~~~~~~~~~~~~~~~~~~~~~   Loss curves --------------------------------------------------------------------------- We can observe that the DCGAN is making progress as evidenced by the decreasing losses of both the discriminator and generator. An intriguing observation is that when DiffAugment is employed, the discriminator loss is higher compared to when it's not used, while the generator loss is lower. This phenomenon is likely attributable to the improved regularization of the discriminator enabled by differentiable augmentation. Notably, there isn't a substantial contrast in loss curves between the basic and deluxe augmentation strategies.   Selected samples --------------------------------------------------------------------------- Below we show some example samples from beginning, middle and end of training, with Deluxe data augmentation and DiffAugment.     Cycle GAN =========================================================================== In CycleGAN, we convert images from two domains, $X$ and $Y$, into each other's domain, instead of generating images from random noises. To do that we have two generators, $G_{X \rightarrow Y}$ and $G_{Y \rightarrow X}$, and two discriminators, $D_X$ and $D_Y$. As the model has two generators and two discriminators, we have two adversarial losses, one for each generator. Additionally, we have two cycle-consistency losses, one for each generator. The adversarial loss is the same as in DCGAN, and the cycle-consistency loss is the L2 loss between the input and the output of the generator.  Early results --------------------------------------------------------------------------- I first train grumpifyCat with only 1000 iterations for sanity check, with and without cycle consistency loss. Cycle GAN at itr=1000, without cycle consistency loss   Cycle GAN at itr=1000, with cycle consistency loss   Extended results --------------------------------------------------------------------------- ### Cats Cycle GAN at itr=10000, without cycle consistency loss   Cycle GAN at itr=10000, with cycle consistency loss   ### Apples and Oranges Cycle GAN at itr=10000, without cycle consistency loss   Cycle GAN at itr=10000, with cycle consistency loss   Obeservations --------------------------------------------------------------------------- Implementing cycle consistency loss appears to enhance the quality of generated images, resulting in sharper details and reduced artifacts, particularly noticeable in datasets like apples and oranges. Additionally, it introduces more diversity in outcomes, which is particularly evident in datasets like cats. The enhancements are more pronounced in the fruits dataset, likely due to its greater complexity with diverse backgrounds compared to the cats dataset, which primarily focuses on facial features in each sample. DCDiscriminator vs PatchDiscriminator --------------------------------------------------------------------------- Up to this point, our outcomes have relied on the PatchDiscriminator, which assesses the authenticity of individual image patches. However, in this segment, we adopt the DCDiscriminator, which evaluates entire images as real or fake. Compared to the PatchDiscriminator approach, these results exhibit a slight decrease in sharpness. Notably, the generated Russian Blue cats appear more blurred, and the generated oranges contain more artifacts than their PatchDiscriminator counterparts. Nevertheless, the disparity between the two approaches is subtle. The PatchDiscriminator methodology ensures that each segment of the generated image appears authentic, whereas the DCDiscriminator focuses on validating the overall authenticity of the entire image. This shift in focus may result in subsections of the image appearing less authentic. Below are results with DC Discriminator with cycle consistency loss trained for iterations = 10000 ### Cats   ### Apples and Oranges   Bells and Whistles =========================================================================== Pre-trained diffusion model --------------------------------------------------------------------------- I use the prompt "a grumpy cat" to generate images using the diffusion model, stabilityai/stable-diffusion-2-1  Train a diffusion model --------------------------------------------------------------------------- I train a UNet2DModel from sctach for 200 epochs. Below are the results at different iterations