Brief introduction
In this project, we are implementing two variations of GANs, DCGAN and CycleGAN, to get some hands-on experience of training and testing GANs.
DCGAN
DCGAN is a deep convolutional GAN, which is a direct extension of GAN. The generator maps some random noise to an image, and the discriminator tries to distinguish the generated image from the real one.
CycleGAN
Unlike in DCGAN where we have a single image domain and try to reconstruct an image, in CycleGAN, we have two image domains(say, grumpy cats and blue cats), and we try to map the image into the other domain. That means we should get a grumpy cat from a blue cat, and a blue cat from a grumpy cat. We have two generators and two discriminators. The generators try to map the image from one domain to the other, and the discriminators also try to distinguish the generated image from the real one.
Deep Convolutional GAN
In DCGAN, the generator is a series of transposed convolutional layers, and the discriminator is a series of convolutional layers. The generator takes a random noise as input, and the discriminator takes an image as input. The loss function coincides with the following algorithm, which is mean squared error loss. The training process is pretty straightforward. We first train the discriminator, and then train the generator in a single training iteration. We repeat this process until the loss converges or after adequate loops(6000~10000).
Given kernel size = 4, stride = 2, we need to decide the padding for the following architecture:
I followed the equation: \(o = \frac{i + 2p - k}{s} + 1\), where \(o\) is the output size, \(i\) is the input size, \(p\) is the padding, \(k\) is the kernel size, and \(s\) is the stride. Also, notice that \(o = \frac{i}{2}\). In this case, \(p = 1\).
For generator, it is said that it's best not to apply upsampling. I simply used nn.ConvTranspose2d. The related formula is \(o = s(i-1) + k - 2p\). In the first layer, \(i = (1, 1)\), \(o = (4, 4)\), we could give \(p = 0\), \(k = 4\). \(s\) could be any since \(s(i-1)\) is always 0. I picked \(s = 1\) anyway.
The rest layers are implemented with up_conv function. It first upsample the input and then apply a convolutional layer. Assume that we want to set upsample factor to 2, we should maintain the size during conv layer. That is, for \(o = \frac{i + 2p - k}{s} + 1\), let \(o = i\), we could pick \(p = 1\), \(k = 3\), \(s = 1\) to satisfy the requirements.
Results
Note: I used extra vertical flip augmentation for DCGAN, while for CycleGAN, I didn't. I guess it created extra difficulty for the generator, and for the following 4 cases, basic preprocess, basic plus diff augementation, deluxe, deluxe plus diff augementation, generator in basic preprocess didn't produce a correct result. The other 3 are all successful at least in generating something looks like a cat.
loss curves
The loss curves are as follows. It could be observed that, losses are relatively low. However, we could hardly find any loss that converges stably and progressively towards zero. This is because this is an adversarial training process, where generators and discriminators compete with each other. It's reasonable that the losses fluctuates up and down. Another observation is that, when diffaug is on, the discriminator makes more effort to distinguish the generated image from the real one. This is because it should learn the differences between artifacts and diff augemented features.
--data_preprocess=deluxe
--data_preprocess=deluxe --use_diffaug
When using deluxe data preprocess without diffaug, my model didn't perform well. From the result picture, I infer that the vertical augementation might have created difficulty for the generator to learn the features of the cat and generate either a upright or upside-down cat, and the dataset was not large enough. On the contray, deluxe with diffaug output better results. However, I noticed that the upside-down image is not good enough, therefore, I disabled vertical flip augmentation for the following experiments.
CycleGAN
In CycleGAN, we have two generators and two discriminators. The generators try to map the image from one domain to the other, and the discriminators also try to distinguish the generated image from the real one. The loss function is similar to DCGAN, as shown in the following figure. The training process is a bit more complicated than DCGAN. We first train the discriminators, then train the generators, and finally train the discriminators again. We repeat this process until the loss converges. I also implemented cycle consistency loss with MSE loss. Cycle consistency loss is used to ensure that the generated image could be mapped back to the original image, so that the generator could generates an output that has more resemblance to the original image.
Results from early iteration
Note: The results are using patch discriminator, until the part of comparison between dc discriminator and patch discriminator.
without cycle consistency loss
with cycle consistency loss
Final results - cats
disadvantage of cycle consistency loss
As I am using MSE loss for cycle consistency loss, the results with the loss show closer color to the original input. For example, the generated blue cats are lighter compared to the ones from the naive model. At the same time, the generated grumpy cats are darker. I think this could be addressed through adjusting the weight for cycle loss.
advantage of cycle consistency loss
The cycle consistency loss helps preserve the feature location. For example, take a look at blue to grumpy, the second row, the second coloumn. Without cycle consistency loss, the generated cat is back in the middle, while with cycle consistency loss, the generated cat is still on the left side.
DC discriminator vs Patch discriminator
For other constraints: --data_preprocess=deluxe --use_diffaug --use_cycle_consistency_loss
The results from dc discriminator performs worse than patch discriminator. Especially for the picture on the right, some faces of the cats are not recognizable, and are more blurry.
Final results - fruits
disadvantage of cycle consistency loss
Cycle consistency is very likely to be influenced by the local color of the input. For instance, we could take a look at apple to orange: the third row, the second coloumn, and the fourth row, the first coloumn. The cycle consistency loss tries to preserve the color of the leaves, thus generating a weird sample and even had bad impacts on color of the fruit.
advantage of cycle consistency loss
The cycle consistency loss helps lower the saturation to a reasonable level. Due to some background, both approaches didn't generate perfect samples, but the one with cycle consistency loss has a better color balance.
DC discriminator vs Patch discriminator
Comparing with previous results from patch discriminator, the results didn't differ much. Theoretically, patch discriminator captures more local features, while dc discriminator captures more global features. I think it's because our input images are of low resolution, so the two discriminators didn't differ much. Also, unlike the cats, the fruit dataset is more complicated.
Bells & Whistles: Pretrained diffusion model
I played with DreamStudio's pretrained diffusion model. I input my prompts and images with a ratio. It appears to me that a longer, more detailed prompt gives more diverse results. The results are as follows.
High resolution results
I also tried to high resolution images from a given link. The data I used are /high_res_images/grumpifyA256 and /high_res_images/grumpifyB256.
With high resolution images, there is a slightly clearer difference between dc discriminator and patch discriminator. The patch discriminator generates more detailed images, while the dc discriminator generates more blurry images.
Data of my own choice
The dataset is collected using the script from https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.
Cat Animation
This coloumn shows a cat animation generated from samples over time in the training process of DCGAN.