16726 Assignment 3

by Zi Wang (ziwang2)

Project Overview

This project aims to deepen our comprehension of Generative Models through practical experience by developing several well-known architectures, including:

Deep Convolutional GAN (DCGAN): This model adheres closely to the initial GAN concept, which involves transforming noise into realistic images. We employ DCGAN here to create lifelike images of grumpy cats.

CycleGAN: This model enables us to explore the application of GANs in converting images from one category (such as grumpy cats) to another (like Russian blue cats) and back. In my view, it's a more fascinating use of GAN technology.

DDPM (Denoising Diffusion Probabilistic Models): This is the most popular architecture in generative models. A lot of powerful tools, like Stable Diffusion, Stable Video Diffusion, etc., are based on this work.

DCGAN

I implemented a convolutional GAN aimed at producing images of grumpy cats. This involved building a discriminator network and a generator network, finishing a training loop, and enhancing the outcome quality through input data augmentation.

Padding

Building the discriminator and generator networks was relatively straightforward, thanks to the guidance provided in the assignment's documentation. However, a critical aspect to consider is the necessity of applying appropriate padding during the upsampling and downsampling processes in the convolutional layers to achieve the desired output size for each layer. To calculate the output image size, I used the following formula:

output_size = floor(input_size + 2 * padding - kernel_size) / stride + 1

For instance, in the first layer of the discriminator, with input_size = 64, output_size = 32, kernel_size = 4, and stride = 2, the required padding is P = 1.

Loss Curves

Here are some images of the Generator and Discriminator losses given different augmentation procedures.

	Generator	Discriminator
basic
deluxe

We see that:

(1) Differentiable augmentation, evidently outperforms as it enables the generator to attain a reduced loss, while making it more challenging for the discriminator to distinguish between real and fake images (higher loss).

(2) When employing advanced preprocessing techniques, there's a noticeable improvement in performance, characterized by a decreased loss for the generator and a somewhat increased loss for the discriminator.

(3) For a well-trained GAN, neither the generator nor the discriminator should "win" outright. Instead, their losses should indicate a balance, with both networks improving over time.

Samples

Below are some sample images of the output of this algorithm given different augmentation procedures.

	basic	basic+diffaug	deluxe	deluxe+diffaug
iter=200
iter=6400

It is evident that the use of both deluxe and differentiable augmentations markedly enhances the quality of the outputs. A notable transformation observed with more training iterations is the emergence of more detailed features in the output images. Initially, the images of cats might appear blurry, but as training progresses, the images evolve to include crisp details like whiskers and the texture of fur.

CycleGAN

In this section, I implemented CycleGAN for transforming images between different classes. This process was achieved by employing a discriminator for each class, notably patch discriminators that evaluate segments of the input image as real or fake. While a GAN structured in this manner can already execute transformations across classes, I further incorporated a cycle-consistency loss term. This addition aids in guaranteeing that converting from class X to class Y is effectively the reverse of converting from class Y to class X.

Here are some results:

--disc patch --train_iters 1000

X to Y	Y to X

--disc patch --use_cycle_consistency_loss --train_iters 1000

X to Y	Y to X

--disc patch --train_iters 10000

X to Y	Y to X

--disc patch --use_cycle_consistency_loss --train_iters 10000

X to Y	Y to X

--disc dc --use_cycle_consistency_loss --train_iters 10000

X to Y	Y to X

--disc patch --train_iters 1000

X to Y	Y to X

--disc patch --use_cycle_consistency_loss --train_iters 1000

X to Y	Y to X

--disc patch --train_iters 10000

X to Y	Y to X

--disc patch --use_cycle_consistency_loss --train_iters 10000

X to Y	Y to X

--disc dc --use_cycle_consistency_loss --train_iters 10000

X to Y	Y to X

Cycle Consistency Loss:

According to my observations, there are mainly two pros of Cycle Consistency Loss:

1. Improved Quality of Inverse Transformations: With cycle consistency loss, the model ensures that an image converted from class X to class Y can be transformed back to a version of the original image in class X. This cycle helps maintain key features and reduces artifacts, leading to higher quality transformations.

2. Preservation of Content: The cycle consistency loss encourages the model to preserve the content of the original image while changing only domain-specific characteristics. Without this loss, the model might focus solely on fooling the discriminator, sometimes at the expense of losing important features of the input image.

DC Discriminator

In the absence of a patch-based discriminator, the translated images tend to appear grainier and more blurred, which aligns with our expectations. The primary difference between the DCDiscriminator and the PatchDiscriminator lies in their approach to analyzing images. The DCDiscriminator considers the image globally, which encourages the generation of images that are internally consistent but might sacrifice some level of detail. On the other hand, the PatchDiscriminator assesses the image in smaller segments, allowing it to focus on the intricacies and textures of specific areas. This localized assessment encourages the generation of images with higher detail and texture complexity.

Bells & Whistles (Extra Points)

Implement and train a diffusion model on our datasets. (10 pts)

I implemented the UNet, training schedule, sampling methods, etc., to train a naive diffusion model. Please refer to code for further details.

python train_ddpm.py

python tset_ddpm.py

Here are some output results.

Generate samples using a pre-trained diffusion model. (5 pts)

I used Stable Diffusion 2.1 to generate cats with prompts: 1. A grumpy cat and 2. A russian blue cat.

Here are some results.