In this assignment, we explore GAN (generative adversarial network) to synthesis and generate images of cat using some training images of cat. We try two methods called DC-GAN and cycle gan to achieve our goal and in the end also try diffusion models for the same tasks. We use 2 datasets i.e. cat dataset and apples and oranges dataset.
DC-GAN is a type of the GAN architecture that uses deep convolutional neural networks in its generator and discriminator architecture. It consists of two networks: a generator and a discriminator. The generator takes random noise as input and generates images, while the discriminator tries to distinguish between real images from the dataset and fake images generated by the generator. Both networks are trained simultaneously in a minimax game framework, where the generator tries to produce more realistic images to fool the discriminator, and the discriminator improves its ability to distinguish real from fake images.
In the downsample method in the discriminator, we have to downsample the image by a factor of 2. We are given that we will use a kernel of size 3 and since we are downsampling the stride will be 2. Then the paddng can be calcualted using the formula:
In the above formula for the first conv, H_in = 64, H_out = 32, dilation = 1, kernel size = 4 and stride = 2. Hence according to the forumula above, padding = 1. It can be done similaraly for all the intermediate layers.
| GAN curve | Discriminator curve |
|---|---|
|
|
| GAN curve | Discriminator curve |
|---|---|
 |
 |
| GAN curve | Discriminator curve |
|---|---|
 |
 |
| GAN curve | Discriminator curve |
|---|---|
 |
 |
From the above curves we can see that the Discriminator loss curve decreases, indicating that the Discriminator is improving in distinguishing real images from fake ones. The generator performance also starts from a high value and decreases as the training progresses. This implies that generator gets better at producing fake images and is able to fool the discriminator as the training progresses. Eventually as both the model converge, we can see that the discriminator has similar loss for both real and fake images and is not able to distinguish them and hence should have an accuracy of 50% on the mixed data.
Another thing that we can see that is the with deluxe data pre-processing the discriminator loss is comparitively higher, hence it is more challenging for the discriminator to learn with deluxe pre-processing as there more data for the discriminator to train on and this also prevents overfitting due to less data. Hence this acts as a regularization and imrpoves the results.
| Basic Processing | Deluxe Processing | |
|---|---|---|
| Without Diffrentaible Augmentation |  |
 |
| With Diffrentaible Augmentation |  |
 |
From the first row of the above table we can see that with basic processing only, there is a mode collapse on generator results. This indictes that there was not enough data and regularization for the network to train and the generator can learn the data distribution.
However, when we add deluxe processing, then the results are much better and mode collasp problem is not there in the generated results. This means that augmentation is very neccessary when we have less data. Hence deluxe augmentation enhances the generated results.
From the second row of the above table we can see that with diffrentiable augmentation the results further imrpove and are more realistic. From last column we can see that without differentaible augmentation, there is slight artifacts in the generated results, but with differentaible augmentations, there are no artifacts. This proves the importance of differentaible augmentation.
Results for iteration = 200 with deluxe pre-processing and diffrentiable augmentation are as defined below:
Results for iteration = 6400 with deluxe pre-processing and diffrentiable augmentation are as defined below:
The samples at iteration 200 are blurry and dont resemble the cat images properly and have lot of noise. The samples at 6400 are much clean and have more frequency details as well. They resemble cat images much better than samples at 1000.
CycleGAN is a type of generative adversarial network (GAN) used for image-to-image translation tasks. The key innovation of CycleGAN is the introduction of cycle consistency, which ensures that the translated images remain faithful to the original content. This is achieved by enforcing the property that translating an image from domain A to domain B and then back to domain A should yield an image similar to the original.
CycleGAN consists of two generators and two discriminators. The generators learn the mappings between the two domains, while the discriminators differentiate between real and translated images. During training, the generators aim to minimize both adversarial loss and cycle-consistency loss.
The initital results of Cycle GAN with patch discriminator, deluxe pre-processing without cycle-consistency at iteration=1000 are:
| X->Y | Y->X |
|---|---|
 |
 |
The initital results of Cycle GAN with patch discriminator, deluxe pre-processing with cycle-consistency at iteration=1000 are:
| X->Y | Y->X |
|---|---|
 |
 |
The final results of Cycle GAN with patch discriminator, deluxe pre-processing without cycle-consistency at iteration=10000 are:
| X->Y | Y->X |
|---|---|
 |
 |
The final results of Cycle GAN with patch discriminator, deluxe pre-processing with cycle-consistency at iteration=10000 are:
| X->Y | Y->X |
|---|---|
 |
 |
The final results of Cycle GAN with patch discriminator, deluxe pre-processing without cycle-consistency at iteration=10000 are:
| X->Y | Y->X |
|---|---|
 |
 |
The final results of Cycle GAN with patch discriminator, deluxe pre-processing with cycle-consistency at iteration=10000 are:
| X->Y | Y->X |
|---|---|
 |
 |
From the above results we can see that the resutls are much better with cycle consistency loss. The results with cycle consistency loss are more realistic and have less artifacts compared to results without cycle consistency loss. Cycle consistency loss ensures that the information content of an image remains intact after being translated between domains. This is vital for maintaining the semantic meaning and structural integrity of the images during the translation process. By enforcing cycle consistency, CycleGAN prevents the loss of important details that might occur in the absence of such a constraint.
Incorporating cycle consistency loss also acts as a form of regularization during training, helping to prevent overfitting and improve generalization. By enforcing consistency between the original and reconstructed images, the model learns a more robust mapping between the two domains. This regularization helps to mitigate issues such as mode collapse and improves the overall stability of the training process.
The final results of Cycle GAN with DC GAN discriminator, deluxe pre-processing with cycle-consistency at iteration=10000 are:
| X->Y | Y->X | |
|---|---|---|
| DC GAN discriminator |  |
 |
| Patch GAN discriminator | |
|
| X->Y | Y->X | |
|---|---|---|
| DC GAN discriminator |  |
 |
| Patch GAN discriminator | |
|
From the above results we can see that patch gan is able to produce much better results than DC Gan. For example for the cat dataset Y->X transitiion we can see that DC GAN results have more artifiacts and are not able to create realistic cats. We can observe this behaviour in apples-to-orange dataset as well. Hence Patch GAN is able to produce better results.
This happens because PatchGAN divides the generated and real images into small patches and classifies each patch individually. This allows for more localized discrimination, providing finer control over the details of the generated image. PatchGANs are particularly effective in generating high-resolution images with fine textures and details. On the other hand, DCGAN typically uses a discriminator that evaluates the entire image at once. While this approach captures global features and structures, it might overlook finer details and textures, especially in high-resolution images.
I wrote code to train a diffusion model with UNet architecture.
After training for 2000 epochs, I then sampled data using diffusion model and the results are:
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Hence we can see that the diffusion model can generate realistic results.