16-726 Assignment 3

Overview

In this assignment, we are training a DCGAN to generate cat images and a CycleGAN to convert between different cat breeds and between apples and oranges. Different augmentations, architectures, and loss functions are explored and compared.

Implement the Discriminator of the DCGAN

The output size \(O\) along a given dimension with input size \(I\) and padding \(P\) on each side is given by: \[ O = \left\lfloor \frac{I + 2P - K}{S} \right\rfloor + 1, \] where \(O = \lfloor I/2 \rfloor,\, K = 4,\, S = 2\), hence \(P = 1\).

Differentiable Augmentation

Differentiable Augmentation. Top left: basic. Top right: basic with differentiable augmentation. Bottom left: deluxe. Bottom right: deluxe with differentiable augmentation. All sampled at 6400 iterations.

Regular augmentation is only applied to the real images but not the generated ones, hence the generated images are less diverse and might be memorized by the discriminator. Differentiable augmentation is applied to both real and generated images, which requires the augmentation to be differentiable so that the gradients can be backpropagated through it to the generator.

Comparing the results, no augmentation leads to mode collapse, while differentiable augmentation tends to generate images with more realistic colors and fewer artifacts compared to the deluxe transformation.

Experiment with DCGANs

When the GAN manages to train, the discriminator loss will decrease and the generator loss will first decrease and then increase. This is because when the generator continues to improve, it will be harder for the discriminator to distinguish the generated images from the real ones, hence the discriminator loss will increase and the generator loss will decrease, until both converge to some equilibrium.

In the following curves, the discriminator total losses for the ones without differentiable augmentation continues to decrease and are lower than the ones with differentiable augmentation. On the contrary, the generator losses for the ones with differentiable augmentation are lower than the ones without. This matches with the qualitative results above and our expectation about the curves. It also verifies the effectiveness of differentiable augmentation.

Losses of the DCGAN. Discriminator and generator losses under different augmentations. Time weighted exponential moving average is used to smooth the curves.

Comparison between earlier and later samples. The left one is sampled at 200 iterations and the right one is sampled at 6400 iterations.

Samples at 200 iterations are basically noisy color patches, while samples at 6400 iterations are closer to the real images, despite having low diversity and some artifacts. The later samples have almost not random colors that are out of the distribution of the real images, and the texture and structures are also closer to the real images. The samples also become much sharper during the training process.

CycleGAN Experiments

Training for 1000 iterations

Results at 1000 iterations without cycle consistency loss.

Results at 1000 iterations with cycle consistency loss.

Training for 10000 iterations

Results at 10000 iterations without cycle consistency loss.

Results at 10000 iterations with cycle consistency loss.

Apple to Orange

Results at 10000 iterations without cycle consistency loss.

Results at 10000 iterations with cycle consistency loss.

Cycle consistency loss gives better results on both the cat and the apple to orange datasets. On the cat dataset, results with cycle consistency have poses closer to the original images and sometimes more reasonable textures. On the apple to orange dataset, results with cycle consistency have more natural colors (fewer false color artifacts) and more realistic structures. The results without cycle consistency can sometimes have unnatural colors and blurry details.

Cycle consistency loss helps to enforce the mapping to be bijective, which implicitly penalizes the generator for generating images that are too different from the original images. This might explain why the results with cycle consistency loss have fewer artifacts, as they preserve the structure and colors of the original images better.

Patch and DC Discriminator

Results at 10000 iterations with DC discriminator on the cat dataset.

Results at 10000 iterations with DC discriminator on the apple and orange dataset.

When the original DC discriminator is used, the results generated on the cat datasets contain some color artifacts at local regions, which are not present in the results from the patch discriminator. However, DC discriminator's results seem more coherent globally, while the patch discriminator's results sometimes have regions that are not well aligned with each other, e.g., the two eyes of the cats. On the apple to orange dataset, the results from the DC discriminator have less natural colors and blurrier local details compared to the results from the patch discriminator.

This might be because the DC discriminator predicts over the entire image and thus incoherent local regions are easier to be detected and predicted as fake. The patch discriminator, on the other hand, predicts over overlapping patches and thus the results may have better local details but worse global coherence. Predicting over the entire image might also be easier than predicting individual patches, hence the DC discriminator may overfit to the dataset more compared to the patch discriminator, which explains the color and texture artifacts in the apple to orange results.

Bells & Whistles

Train a Diffusion Model

We implement a simplified version of the diffusion model based on the repo. The generated samples are shown below. The model is trained for 2000 epochs.