Assignment #3 - Cats Generator Playground

Yu-Chen Lin

I. Description

In this homework, we try to synthesis new images based on the given images. We aim to train two models to handle two different kinds of tasks. The first one is DCGAN. It is used to generate images based on one specific domain. The other one is CycleGAN. It can transfer the style from one domain to another given the paired images from two domains.

II. DCGAN

Padding

The derivation of padding is shown as follows.

Training Loss (--data_preprocess=basic)

The left one is the discriminator loss, and the right one is the generator loss. The red curve is with DiffAugment, and the orange curve is without DiffAugment.

The loss difference between the discriminator and the generator should not be too far away. For example, if the discriminator domininates the generator by classifying all the generated images as fake, then the generator may not improve over time. We can clearly see that the D and G loss curves with and without DiffAugment are both reasonable.

For the case with DiffAugment, the loss difference between the discriminator and the generator is even closer. It means they are competing with each other more seriously, which may lead to stronger discriminator and generator.

Training Loss (--data_preprocess=deluxe)

The left one is the discriminator loss, and the right one is the generator loss. The blue curve is with DiffAugment, and the green curve is without DiffAugment.

The change of loss for the discriminator and the generator is not too abrupt (but smooth) for both cases. It implies that the two modules are competing with each other. We also can observe for the blue curve, the generator struggles a little bit by increasing the loss for the iteration interval 4000~5000.

Just like the previous case, we can also observe the loss difference between the discriminator and the generator is closer in the configuration with DiffAugment, which may imply better convergence points.

Samples with --data_preprocess=deluxe and DiffAugment

The left one shows the samples after only 200 iterations, and the right one demonstrates the samples after 6400 iterations.

The original samples are very blurry and not easy to be identified as cats.

The samples improve through depicting the contour or sketch of what cat looks like. For example, at first we cannot identify where the eyes and the nose are, but for the final samples, we can clearly see a more complete illustration of the eyes and the nose.

III. CycleGAN

Patch Discriminator with 1000 iterations on grumpifyCat dataset

Patch Discriminator with 10000 iterations on grumpifyCat dataset

We discuss the effect of the cycle consistency loss for grumpifyCat as follows.

For the positive effect, it can preserve the overall shape when transferring from the source domain to the target domain. For example, for Y to X, without the cycle consistency loss, we cannot identify the image of (row, col) = (2, 7). On the other hand, the shape of the image is preserved can the generated image with the cycle loss.

For the negative effect, it can sometimes overfit to the background. To see this on X to Y images, please look at the image of (row, col) = (1, 1). The image without the cycle loss have eyes showing, while the image with the loss seems to fit into the background so that the eyes cannot be shown completely.

Patch Discriminator with 10000 iterations on apple2orange dataset

We discuss the effect of the cycle consistency loss for apple2orange as follows.

For the positive effect, it can make the generated images more smooth and thus reasonable. For example, for the X to Y case, the image at (row, col) = (1, 3) transform to a more realistic face by using the cycle consistency loss.

For the negative effect, the smoothness effect seems to sacrifice some diversity on the image. To see this, we can focus on the image at (row, col) = (1, 7) for the Y to X case. The original image without the cycle loss can produce an apple with the green part (that the apple has not been ripe yet). On the other hand, the smoothing effect caused by the cycle loss make this realistic part disappear.

DC vs Patch Discriminator on grumpifyCat dataset

Patch discriminator can generate local pixels more effectively. For example, the eyes generation is more reasonable when using the patch discriminator; see X to Y, the images at (row, col) = (1, 7) or (2, 7) for failure examples using the DC discriminator.

DC vs Patch Discriminator on grumpifyCat dataset

The generative content using the DC discriminator seems to be more messy; see Y to X, the images at (row, col) = (1, 3) or (1, 7). On the other hand, the generative images using the patch discriminator can produce the local pixels more successfully as the results look smoother.

IV. Bells & Whistles

Diffusion Model

For diffusion model, I adapted the code from The Annotated Diffusion Model. Below are the demonstration of the generation results. The results look more realistic than the previous models.

Pre-trained Diffusion Model

I used Promptist to optimize the prompt and applied runwayml/stable-diffusion-v1-5 to do the generation.

a grumpy cat thinks about the endless homework and tests, by greg rutkowski, digital art, realistic painting, fantasy, very detailed, trending on artstation.

A guy choosing between apples and oranges to eat, by Stanley Artgerm Lau, WLOP, Rossdraws, James Jean,

Variational Autoencoder

Below are the results from different iterations. The results of VAE seem to be blurry, which is under expectation.

Generative Models Zoo!