In this assignment, I first implemented and trained a Deep Convolutional GAN (DCGAN) to generate Grumpy cat images from samples of random noise. This involved implementing the discriminator and generator networks in PyTorch, creating the training loop, and implementing differentiable augmentation. Then, I implemented and trained a CycleGAN to perform unpaired image-to-image translation between Russian-Blue cats and Grumpy cats. I also trained a CycleGAN to perform unpaired image-to-image translation between apples and oranges.
The formula to calculate the output size of a convolutional layer is given by: \[ \text{output size} = \frac{\text{input size} - \text{kernel size} + 2 \times \text{padding}}{\text{stride}} + 1 \] Given that we use kernel size 4 and stride 2, we find that the padding size should be 1 when we want to downsample by a factor of 2: \[ H = \frac{2H - 4 + 2 \times \text{padding}}{2} + 1 \] \[ H = H - 2 + \text{padding} + 1 \] \[ \text{padding} = 1 \] For conv5, where we downsample by a factor of 4, the padding size is 0: \[ 1 = \frac{4 - 4 + 2 \times \text{padding}}{2} + 1 \] \[ 1 = 2 - 2 + \text{padding} + 1 \] \[ \text{padding} = 0 \]
The figure below illustrates the DCGAN discriminator architecture (also defined in the assignment instructions):
The figure below illustrates the DCGAN generator architecture (also defined in the assignment instructions):
up_conv1, I used a ConvTranspose2D layer with kernel size 4, stride 1, and padding 0.
Effect of differentiable augmentation: The deluxe data preprocessing applies image augmentations to only the real images during training. Differentiable augmentation applies image augmentations to both the real AND fake images during training, therefore reducing overfitting in both the generator and the discriminator. In general, the generator loss becomes lower than it was without differentiable augmentation. Therefore the discriminator has a harder time differentiating between real and fake images, and the discriminator loss increases.
What the loss curves should look like: If the GAN manages to train successfully, the generator loss should trend downwards and the discriminator loss should trend upwards as the number of training steps increases. It makes sense that the loss curves are not smooth because optimizing the generator vs. discriminator are adversarial objectives - as the generator gets better, the discriminator gets fooled more.
Sample quality: Training with differentiable augmentation seems to have improved the quality of the generated images at 6400 steps. The images have less unrealistic artifacts and have more defined features. In general, as training progresses, sample quality improves, from very blurry blobs of grumpy cat colors to recognizable images of grumpy cats.
Effect of Cycle-Consistency Loss: In general, Cycle-Consistency Loss improves the quality of the generated images. The samples look more realistic and have less artifacts. The CycleGAN also converges faster - at 1,000 iterations, the samples already have fewer white artifacts compared to the samples without Cycle-Consistency Loss. The image translation result is also better: the generated cat images have more accurate spatial placement of facial features. Cycle-Consistency Loss's improvements make sense because it trains a GAN for both directions of image translation, so when cycle-consistency loss is applied, the generated samples are encouraged to stay within one of the 2 image domains and therefore look more realistic.
Effect of PatchDiscriminator vs. DCDiscriminator: The PatchDiscriminator seems to generate better image translation results than the DCDiscriminator. This is mostly evident in the Grumpy cat to Russian Blue cat translation, where the PatchDiscriminator preserves the structure of the input image better while the DCDiscriminator seems to only generate a front-facing Russian Blue cat. The PatchDiscriminator's improvements make sense because it discriminates image patches rather than the entire image, making it more effective at capturing local image structures.
I used the provided skeleton code and referenced https://huggingface.co/blog/annotated-diffusion to implement and train a simple diffusion model on our Grumpy Cat dataset. The generated results look a lot more like real photos of grumpy cats than the DCGAN and CycleGAN samples.
I also tried using the diffusers implementation of Stable Diffusion-XL to do text-to-image generation. Note that the original generated image size is 1024x1024 and the samples below are resized for visibility.
