Assignment 3

• Sihan Liu (sihanliu@andrew.cmu.edu)

0. Project Overview

This project includes the following features:

• Section 1: Deep Convolutional GAN.

  • 1.1: Train the DCGAN
  • 1.2: Screenshots of loss and training curves
  • 1.3: Visualization of samples with deluxe data preprocessing and differentiable augmentation enabled

• Section 2: CycleGAN.

  • 2.1: Train the CycleGAN w/ and w/o the cycle-consistency loss
  • 2.2: Train CycleGAN 10000 iterations
  • 2.3: Train on the apple2orange dataset
  • 2.4: Difference between the results with and without the cycle consistency loss
  • 2.5: Train the CycleGAN with the DCDiscriminator for comparison

• BELLS & WHISTLES:

  • I trained the diffusion model on the provided cat dataset and inference with DDPM
  • I used the current off-the-shell pretrained Stable Diffusion 2.0 model to generate cat images for our task
  • I also implemented spectral normalization on GANs for stability

1. Deep Convolutional GAN

1.1

In discriminator, for each convolutional layer, given kernel size K = 4 nd stride S = 2, we can calculate the padding:

(N + 2 * P - K)/ S + 1 = N/2

(N + 2 * P - 4)/ 2 + 1 = N/2

Then we can get P = 1

In generator, for the first layer (up_conv1), it is better to directly apply convolution layer without any upsampling to get 4x4 output. To implement this, I used K = 4, S = 1, P = 0 using nn.ConvTranspose2d as the first layer. For the resting layers, I used function up_conv with K = 3, S = 1, P = 1, scale_factor = 2.

1.2 Screenshots of loss and training curves

Screenshots:

deluxe	deluxe + diffaug	basic	diffaug + diffaug

Curves:

Model	deluxe	deluxe + diffaug	basic	diffaug + diffaug
discriminator
generator

Comments:

If GAN manages to train, the loss of discriminator should decrease and the loss of generator increase in general.

1.3 Visualization of samples with deluxe data preprocessing and differentiable augmentation enabled

Result:

#Iteration	200	2200	5400
Sample

Comments:

The samples after more iterations (e.g., 5400) has less noise compared to the ones after fewer iterations (e.g., 200). Both techniques help reduce the effects from overfitting of discriminator while training.

2. CycleGAN

2.1 Train the CycleGAN w/ and w/o the cycle-consistency loss

To run the task:

python cycle_gan.py --disc patch --train_iters 1000
python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 1000

	Iteration: 1000 (X->Y)	Iteration: 1000 (Y->X)
w/o cycle-consistency loss
w/ cycle-consistency loss

2.2 Train CycleGAN 10000 iterations

To run the task:

python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 10000

#Iteration	1000	4000	9000
X->Y
Y->X

2.3 Train on the apple2orange dataset

To run the task:

python cycle_gan.py --disc patch --train_iters 1000 --X apple2orange/apple --Y apple2orange/orange
python cycle_gan.py --disc patch --use_cycle_consistency_loss  --train_iters 1000 --X apple2orange/apple --Y apple2orange/orange

	Iteration: 1000 (X->Y)	Iteration: 1000 (Y->X)
w/o cycle-consistency loss
w/ cycle-consistency loss

2.4 Difference between the results with and without the cycle consistency loss

Comments:

According to the comparisons of samples in Section 2.1 and 2.3, we observe that training with cycle consistency loss can help generate images that maintain most of the details from the input images. This difference may because of the extra supervision of the initial input images while translating.

2.5 Train the CycleGAN with the DCDiscriminator for comparison

To run the task:

python cycle_gan.py --disc dc --use_cycle_consistency_loss  --train_iters 1000
python cycle_gan.py --disc dc --use_cycle_consistency_loss  --train_iters 1000 --X apple2orange/apple --Y apple2orange/orange

Cat Dataset:

	Iteration: 1000 (X->Y)	Iteration: 1000 (Y->X)
patch
dc

Apple2Orange Dataset:

	Iteration: 1000 (X->Y)	Iteration: 1000 (Y->X)
patch
dc

Comments:

According to the samples above, we find that training GAN with patch discriminator can generate better details based on the input image. This may because the patch-level supervision of patch discriminator can better capture the details of images.

3. Bells & Whistles

1. I trained the diffusion model on the provided cat dataset and inference with DDPM. I generated 25 samples and here is the result:

2. I tried to used pretrained stable diffusion v2 on the cat dataset. Here are some results:

"a grumpily cat in real image"	"a grumpily cat in real image"	"a grumpily cat in real image"	"a grumpily cat in real image"

3. When set the parameter use_spectral_norm=True in models, I implemented spectral normalization on GANs by using torch.nn.utils.spectral_norm.