The goal of this assignment is to generate cats using GANs models. In addition to training a DCGAN to generate grumpy cats from random noise, we will also train a CycleGAN to translate between Grumpy and Russian Blue as well as apples and oranges.
After this, I implemented iterative "bells and whistles" improvements:
For this task, I do the following:
The assignment instructions mentions we should include our calculations for determining the padding. From the PyTorch documentation, we get the formula $$H_{out} = \left\lfloor \frac{H_{in} + 2 \times \text{padding}[0] - \text{dilation}[0] \times (\text{kernel\_size}[0] - 1) - 1} {\text{stride}[0]} + 1 \right\rfloor $$ $$32 = \left\lfloor \frac{64 + 2 \times P - (1) \times (4 - 1) - 1} {2} + 1 \right\rfloor $$ $$32 = \left\lfloor \frac{64 + 2 \times P - (1) \times 2} {2} + 1 \right\rfloor $$ $$32 = \left\lfloor 32 \times P - 1 + 1 \right\rfloor $$ $$32 = \left\lfloor 32 \times P \right\rfloor $$ $$1 = P$$ Width is the same thing: $$W_{out} = \left\lfloor \frac{W_{in} + 2 \times \text{padding}[1] - \text{dilation}[1] \times (\text{kernel\_size}[1] - 1) - 1} {\text{stride}[1]} + 1 \right\rfloor $$ $$32 = \left\lfloor \frac{64 + 2 \times P - (1) \times (4 - 1) - 1} {2} + 1 \right\rfloor $$ $$32 = \left\lfloor \frac{64 + 2 \times P - (1) \times 2} {2} + 1 \right\rfloor $$ $$32 = \left\lfloor 32 \times P - 1 + 1 \right\rfloor $$ $$32 = \left\lfloor 32 \times P \right\rfloor $$ $$1 = P$$
The discriminator and generator losses have some similar behavior across all 4 experiments. In particular, they start with higher loss, and then the discriminator becomes better at discovering fake images. From here, there is a divergence in behavior depending on whether the generator succeeds in training.
When the GANs fails to train (like in the first experiment), we can see that the discriminator loss drops quickly and stays very low. It's able to nearly always distinguish between real and fake images. As a result, the generator is unable to learn what direction is better for fooling the discriminator, and it fails to learn a useful distribution.
When the GANs succeeds in training, the discriminator loss still decreases, but not as much. At a certain point, the generator learns to successfully emulate the real images, and the discriminator is unable to distinguish between the two. At this point, both the generator and discriminator loss begins to stabilize and oscillate around a nonzero loss. In the second, third, and fourth experiments, we can see that this kind of trend. We can also see that, generally, the higher the disciminator loss stabilization values, the higher the quality of the generated images.
Experiment 2 shows considerable quality improvement with just differentiable augmentation. There are way fewer artifacts, the fidelity to the cat is much higher, and the cats are much more realistic. We can attribute this to the fact that differentiable augmentation can increase the number of training examples in a trainable meanner.
Experiment 3 shows that the deluxe data preprocessing also improves sample quality, though not nearly as much. There are fewer artifacts, and the cats are a little more realistic. We can attribute this to the fact that augmentations from the deluxe dataloader can increase the number of training examples. However, as the paper Differentiable Augmentation for Data-Efficient GAN Training mentions, this is less effective than their differentiable approach, since the generator will learn a noised distribution rather than the true distribution.
There is noticeable difference betweeen the results with and without cycle-consistency loss. Generally, using cycle-consistency loss seems to produce better quality images. In particular, when translating from Russian Blue to Grumpy cat, the cycle-consistency loss trained GAN produces many believable cats while the non-cycle-consistency GAN produced some very distorted cats.
It makes sense that the model using cycle-consistency loss would produce better results. After all, we provide a loss that encourages a stable/consistent translation between different states to reproduce the same image. As a result, we have a reconstruction loss that is more specific than the vanilla GAN's loss of just belonging to the distribution. In particular, look at the Russian Blue cats to Grumpy cats. Without cycle-consistency, you can often see that the generated Grumpy cat often retains the correct coloring, but has lost a lot of its structural features. These features are preserved with cycle-consistency loss.
For the apples and oranges datasets, we don't see much difference in performance. This may be due to the fact that the shapes of apples and oranges are already fairly close.
Using the DCDiscriminator rather than a patch discriminator shows a degredation in performance. This can be most clearly seen with the cats. The cats are much more blurry and "jaggedy" in appearance. This is because the patch discriminator focuses more on textures and local features, so the generator is forced to make the local features of the object more consistent. As a result, the patch-based cats tend to have a more consistent furry look and are locally consistent.
For apples and oranges, there isn't as large of a difference. This is likely because oranges and apples are smoother and have fewer features than cats. To deal with smoother features, we would need very high resolution images and a patch-based discriminator with focus on very small patches. We also have fewer recognizable features that we can use to judge the quality of the translation.
The following implementation is based off of the works from lucidrains and the annotated diffusion model.
I implemented a fairly standard Unet-based diffusion model. While the baseline code seems to be encouraging a linear beta schedule, from personal experience I found that cosine scheduling seems to almost always work better heuristically, so I just went with that instead.
While training, I noticed that the noise and model seems to have a tendency to converge to uniform shapes rather than an image of some sort.
The model seems to work properly though. I noticed that the initial starting noise may be overly influential on the final output of the model, with little influence from the training dataset. You can see the influence of the noise on the generation in the following figure:
To verify this, I switched to the basic dataloader, limited the number of training samples to the first 10 images of the training dataset, and then reran the model for 100,000 epochs. The results are much stronger.
Even when overfitting though, the images aren't perfect. In particular, they seem to biased towards darker pixels, so the eyes occasionally disappear.
Spectral normalization basically just normalizes each layer. This can mitigate issues with vanishing or exploding gradients. This is a minor code change, but it works really well. My code implementation is based off of this repo which is what papers with code recommended.
