Assignment 3 - Cats Generator Playground

---

Introduction

This assignment includes two parts:

1. Implement a Deep Convolutional GAN (DCGAN) designed to process images. Train the DCGAN to generate grumpy cats from samples of random noise.
2. Implement a CycleGAN for the task of image-to-image translation. Train the CycleGAN to convert between different types of two kinds of cats (Grumpy and Russian Blue), and between apples and oranges.

Part 1: Deep Convolutional GAN

To implement the DCGAN, we need to specify three things: 1) the generator, 2) the discriminator, and 3) the training procedure. We will develop each of these three components in the following subsections.

Implement the Discriminator of the DCGAN

Padding: In each of the convolutional layers shown above, we downsample the spatial dimension of the input volume by a factor of 2. Given that we use kernel size K = 4 and stride S = 2, what should the padding be? Write your answer on your website, and show your work (e.g., the formula you used to derive the padding).

Answer:

$$ \text{new\_size} = \left\lfloor \frac{\text{old\_size} - K + 2 \times \text{Padding}}{S} \right\rfloor + 1 \\ \text{Padding} = \frac{S \times (\text{old\_size} / 2- 1) + K - \text{old\_size}}{2} \\ \text{Padding} = \frac{-2+4}{2} = 1 $$

In the write up, please show results with and without applying differentiable augmentations, and discuss the difference between two augmentation schemes we discussed, in terms of implementation and effects.

Experiment with DCGANs [50 points]

The script saves the output of the generator for a fixed noise sample every 200 iterations throughout training; this allows you to see how the generator improves over time. Include the following in your website:

• Screenshots of discriminator and generator training loss with --data_preprocess=basic, --data_preprocess=deluxe. For each --data_preprocess flag, also show results trained both with and without differentiable augmentation, so you will show 8 curves in total. Briefly explain what the curves should look like if GAN manages to train.

If a GAN manages to train, the curves should flactuate, reflecting a balanced adversarial relationship between the generator and the discriminator.

python vanilla_gan.py  --data_preprocess=basic
python vanilla_gan.py  --data_preprocess=basic --use_diffaug

python vanilla_gan.py  --data_preprocess=deluxe
python vanilla_gan.py  --data_preprocess=deluxe --use_diffaug

	Discriminator Loss	Generator Loss
basic
basic + diffaug
deluxe
deluxe + diffaug

With --data_preprocess=deluxe and differentiable augmentation enabled, show one of the samples from early in training (e.g., iteration 200) and one of the samples from later in training, and give the iteration number for those samples. Briefly comment on the quality of the samples, and in what way they improve through training.

Iteration 200	Iteration 3500	Iteration 6000

Answer: The samples at iteration 200 are blurry and lack any discernable features. By iteration 3500, the samples are starting to resemble cats, but are still blurry and the colors are not well defined. By iteration 6000, the samples are much clearer and resemble cats with defined features and colors.

Part 2: CycleGAN

CycleGAN Experiments [50 points]

Training the CycleGAN from scratch can be time-consuming if you do not have a GPU. In this part, you will train your models from scratch for just 1000 iterations, to check the results.

1. Train the CycleGAN without the cycle-consistency loss from scratch using the command:

python cycle_gan.py --disc patch --train_iters 1000

This runs for 1000 iterations, and saves generated samples in the output/cyclegan folder. In each sample, images from the source domain are shown with their translations to the right. Include in your website the samples from both generators at either iteration 800 or 1000, e.g., sample-001000-X-Y.png and sample-001000-Y-X.png.

• Train the CycleGAN with the cycle-consistency loss from scratch using the command:

python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 1000

Similarly, this runs for 1000 iterations, and saves generated samples in the output/cyclegan folder. Include in your website the samples from both generators at either iteration 800 or 1000 as above.

2. If the previous looks reasonable, it is time to train longer time. Please show results after training 10000 iterations. Include the sampled output from your model.

python cycle_gan.py --disc patch --train_iters 10000

python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 10000

3. Also train on the apple2orange dataset. You can do this by setting the flag --X apple2orange/apple and --Y apple2orange/orange.

python cycle_gan.py --disc patch --train_iters 10000 --X apple2orange/apple  --Y apple2orange/orange

python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 10000 --X apple2orange/apple --Y apple2orange/orange

4. Do you notice a difference between the results with and without the cycle consistency loss? Write down your observations (positive or negative) in your website. Can you explain these results, i.e., why there is or isn’t a difference between the two?

Answer: The results with the cycle consistency loss are much better than the results without the cycle consistency loss. The results without the cycle consistency loss are blurry and lack any discernable features. The results with the cycle consistency loss are much clearer and resemble the source objects with defined features and colors.

The cycle consistency loss helps the model to learn the mapping between the two domains, and helps to ensure that the model is able to generate realistic images. Without the cycle consistency loss, the model may lack the constraint to ensure a consistent mapping, leading to inconsistencies and distortions in the generated images.

5. Train the CycleGAN with the DCDiscriminator for comparison:

python cycle_gan.py --disc dc --use_cycle_consistency_loss

python cycle_gan.py --disc dc --use_cycle_consistency_loss --X apple2orange/apple  --Y apple2orange/orange

Compare and report your observations between the results using DCDiscriminator and PatchDiscriminator. Can you explain the results?

	DCDiscriminator	PatchDiscriminator
cat X_to_Y
cat Y_to_X
apple2orange X_to_Y
apple2orange Y_to_X

Answer: The reults using the DCDiscriminator tend to be more blurry and lack any discernable features compared to the results using the PatchDiscriminator. The PatchDiscriminator is able to capture more local information and is able to generate more realistic images. The DCDiscriminator is a global discriminator, focusing on the overall image. This may lead to capturing less local information and generating less realistic and less varied images.

Bells & Whistles (Extra Points)

Implement and train a diffusion model on our datasets. (10 pts)

python train_ddpm.py 
python test_ddpm.py

The training loss of the diffusion model decreases over iterations, and the results of the diffusion model are shown below:

Generate samples using a pre-trained diffusion model. (5 pts)

Sample 10 images from the trained diffusion model and the results are shown below: