Overview

Part 1: Implementation of Deep Convolutional GAN (DCGAN) to generate grumpy cats from samples of random noise. DCGAN uses a convolutional neural network as the discriminator and a network composed of transposed convolutions as the generator.

Part 2: The second part consists of training Cycle-GAN for the task of image-to-image translation (apples to oranges). The generator in Cycle-GAN first encodes the input via a series of convolutional layers and then transforms the features by passing them through a series of residual blocks. The transformed features are then decoded using transposed convolutional layers.

Part 1: DCGAN

Architecture:

Given the above architecture, the following objective function is used for training: \[ \mathcal{L}_{\text{GAN}}(G, D) = \mathbb{E}_{z \sim p_z(z)}[\log D(G(z))] \] \[ \mathcal{L}_{\text{D}}(D, x, z) = -\mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] - \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \]

Padding Calculation:

For the convolution layers within the discriminator, the padding value is determined by the following expressions:

K refers to the size of the kernel, P represents the padding, and S is the stride.

The dimension of the output can be computed with the equation:

Output Dimension = ⌊ (Dimension of Input - Size of Kernel + 2 * Padding) / Stride ⌋ + 1

For the given data:

32 = ⌊ (64 - 4 + 2 * Padding) / 2 ⌋ + 1

Simplifying the above, we get:

32 = 30 + Padding + 1

Thus, we find that Padding equals 1.

DCGAN Loss Curves:

Results:

Without Diffaug

With Diffaug

The comparison reveals the effectiveness of differentiable augmentations by showcasing comparative loss trajectories when utilized versus not. Incorporating these augmentations significantly lowers generator loss, thus expediting training and improving the generation of high-quality samples.

The enhancement in sample quality throughout training is evident. Initially characterized by vague features, the generated samples progressively display higher definition and clarity, culminating in sharper and more detailed images.

Core to differentiable augmentation, as outlined in the study, is the necessity for augmentations to be differentiable to allow gradient backpropagation to the generator. This method, which augments both real and synthetic samples, diverges from traditional augmentation approaches that modify only real images. The result is a more diverse data distribution and a discriminator that avoids overfitting to real images.

Part 2: Cycle-GAN

Architecture:

CycleGAN is a generative adversarial network (GAN) framework specifically developed for unsupervised image translation tasks that do not require matched training samples. It eschews straightforward image-to-image translation, instead employing a cycle-consistency loss to maintain fidelity in translations when reverted to their original domains.

The optimization of CycleGAN revolves around a composite of objective functions. These include adversarial losses for the generators and discriminators corresponding to each domain, along with a cycle-consistency loss that ensures translation reliability. The aggregate loss, which is the sum of these component losses, is vital for the model's effective training.

L_total(G, F, D_X, D_Y, X, Y) = L_GAN(G, D_Y, X, Y) + L_GAN(F, D_X, Y, X) + λ_cycleL_cycle(G, F)

L_GAN(G, D_Y, X, Y) = E_y∼Pdata(y)[log D_Y(y)] + E_x∼Pdata(x)[log(1 - D_Y(G(x)))]

L_cycle(G, F) = E_x∼Pdata(x)[||F(G(x)) - x||₁] + E_y∼Pdata(y)[||G(F(y)) - y||₁]

Results:

1k Iterations

1k Iterations (with cycle consistency)

10k Iterations

10k Iterations (with cycle consistency)

10k iterations on Fruits Dataset

10k iterations on Fruits Dataset (with cycle consistency)

Advantages: The cycle consistency loss enhances the preservation of input domain characteristics within the translated images, as is markedly evident in the translation outcomes for disgruntled felines.

Disadvantages: However, the cycle consistency loss can create unduly strict correlations between the source and target domains, thereby diminishing the diversity of the resulting images. A notable shortfall is the model's struggle with accurately adopting the target domain's color scheme, often resulting in a tempered average color to ensure cycle consistency. This phenomenon is observable in the top-left example of the apple-to-orange translation showcase.

Using DCDiscriminator

Using DCDiscriminator (with cycle consistency)

In contrast to the PatchDiscriminator that concentrates on localized image segments, the DCDiscriminator's approach to assessing the entire image can lead to a less meticulous retention of the finer details. As a result, the DCDiscriminator tends to produce translations that are not as refined. This becomes slightly noticeable in instances such as grumpy cat images where the hues may not look as natural, and one might detect certain channel artifacts. Despite presumptions of a marked distinction between the two discriminators, their performance difference is relatively indiscernible, particularly in lower resolution images.

Custom Diffusion Model:

For the custom diffusion model, I used StabilityAI's 2.1 SD Model to generate some imaes of a robot:

---

---