Project 1

Brief Overview

This projects aims to practice implementing two GAN models: Deep Convolutional GAN (DCGAN) and CycleGAN. This includes implementing the discriminator, generator modules and training loop. Since the dataset used in this project is small, it is necessary to use some data augmentation methods to prevent overfitting and improve the performance. The augmentation methods used in this project include: RandomCropping, HorizontalFlip, combination of RandomCropping and HorizontalFlip, and I also applied differentiable augmentation (color, translation, cutout) during training. For CycleGAN, I also implemented the Cycle Consistency loss

The following section will elaborate on the details of DCGAN and CycleGAN.

Part 1: DCGAN

Padding: the dimension of after Convolutional Layer can be calculated through the following formula:

Where K is for kernel size, P is for padding, and S is for stride. Thus it is easy to get padding = 1 if we want to reduce the dimension by half with kernel size 4 and stride 2. Thus the padding here should be 1

Generator: I just set the first layer conv layer with kernel = 4, stride = 1 and padding = 3 for the upsampling

The following are the loss curves of the DCGAN experiments

--data_preprocess=basic

--data_preprocess=deluxe

--data_preprocess=basic --use_diffaug

--data_preprocess=deluxe --use_diffaug

All experiments

Both the loss of generator and the discriminator should follow the trend of decreasing as training progressing. Just like the --data_preprocess=deluxe --use_diffaug exp. There may exists some fluctuations in the curve, but the loss should shows a downward trend.

Comparison of two augmentation methods

Obviously, differentiable augmentation improve the performance a lot and help the loss to coverge. The difference between the data augmentation and differentiable augmentation is that: data augmentation only varies the real image, but the differentiable augmentation also varies the sythnesized image. Compared to the data augmentation, differentiable augmentation prevents the discriminator from overfitting to the fake images, thus improve the performance of the discriminator. Under a GAN training scheme, a better discriminator would probably help improve the performance of the generator.

The following are the examples generated after 200 iters and 6400 iters of training, respectively

200 iters

6400 iters

As shown above, the 6400 iter model generated sample has a much better quality comparing to the one generated by the 200 iter model. The noise were reduced and more details of the object(cat) are added to the images during training.

Part 2: CycleGAN

Path Discriminator Structure: I just change the last layer of the DCGAN's discriminator to kernel size = 3, stride = 1 and padding = 1 convolutional layer.

The following sections will show the results and samples from different experiments

Cat Part

python cycle_gan.py --disc patch --train_iters 1000

1000 iters

python cycle_gan.py --disc patch --train_iters 1000 --use_cycle_consistency_loss

1000 iters

Loss Comparison w/o Consistency Loss

python cycle_gan.py --disc patch --train_iters 10000

1000 iters

python cycle_gan.py --disc patch --train_iters 10000 --use_cycle_consistency_loss

1000 iters

Loss Comparison w/o Consistency Loss

Fruit Part

python cycle_gan.py --disc patch --train_iters 1000 --X apple2orange/apple --Y apple2orange/orange

1000 iters

python cycle_gan.py --disc patch --train_iters 1000 --use_cycle_consistency_loss --X apple2orange/apple --Y apple2orange/orange

1000 iters

Loss Comparison w/o Consistency Loss

python cycle_gan.py --disc patch --train_iters 10000 --X apple2orange/apple --Y apple2orange/orange

1000 iters

python cycle_gan.py --disc patch --train_iters 10000 --use_cycle_consistency_loss --X apple2orange/apple --Y apple2orange/orange

1000 iters

Loss Comparison w/o Consistency Loss

There's difference between the results of the two training strategy (w/o Consistency Loss). I think that training without Consistency Loss makes the model better (10k iters, I derived this conclusion based on cat dataset, since they performs roughly same in fruit dataset). Since the images generated by the model trained without Consistency Loss looks more natural (eyes, less noise, less unnatural color blocks)

I think that it might be due to that the Consistency Loss is added at the begining of the training. It's similar to the joint training scheme in speech community like (machine speech chain) which first train a model without the Consistency Loss from scratch until the loss converges, and then add the Consistency Loss to the training loop. Adding Consistency Loss from the begining might cause the model fall into a worse local optimal point (since the loss is larger (show in the loss curve), leading to a bigger step)

Patch vs. DC Discriminator

Cat Part

python cycle_gan.py --disc dc --use_cycle_consistency_loss

1000 iters

python cycle_gan.py --disc patch --train_iters 10000 --use_cycle_consistency_loss

1000 iters

Loss Comparison DC vs Patch

Fruit Part

python cycle_gan.py --disc dc --use_cycle_consistency_loss --X apple2orange/apple --Y apple2orange/orange

1000 iters

python cycle_gan.py --disc patch --train_iters 10000 --use_cycle_consistency_loss --X apple2orange/apple --Y apple2orange/orange

1000 iters

Loss Comparison DC vs Patch

Bells and Whistles:

None