**Learning-Based Image Synthesis** **Assignment 3** - Cats Generator Playground Overview ============================================================== The goal of this assignment is to study GANs. We will be coding and training GANs. The are two parts - DCGAN and CycleGAN. The DCGAN, Deep Convolutional GAN is to generate grumpy cat images from random samples of noise. And the CycleGAN is to perform image-to-image translation between two types of cats and two types of fruits. Deep Convolutional GAN ============================================================== Data Augmentation -------------------------------------------------------------- Basic ```basic_transform = transforms.Compose([ transforms.Resize(opts.image_size, Image.BICUBIC), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), ]) ``` Deluxe ```deluxe_transform = transforms.Compose([ transforms.Resize(osize, Image.BICUBIC), transforms.RandomCrop(opts.image_size), transforms.RandomHorizontalFlip(), transforms.ColorJitter(brightness=0.3, saturation=0.2), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)), ]) ``` Discriminator --------------------------------------------------------------  ``` self.conv1 = conv(3, 32, 4, 2, 1, norm, False, 'relu', spectral_norm=True) self.conv2 = conv(32, 64, 4, 2, 1, norm, False, 'relu', spectral_norm=True) self.conv3 = conv(64, 128, 4, 2, 1, norm, False, 'relu', spectral_norm=True) self.conv4 = conv(128, 256, 4, 2, 1, norm, False, 'relu', spectral_norm=True) self.conv5 = conv(256, 1, 4, 2, 0, norm='', init_zero_weights=False, activ='sigmoid') ```  Generator --------------------------------------------------------------  ``` self.up_conv1 = conv(noise_siz, 256, 2, 1, 2, norm='instance', init_zero_weights=False, activ='relu') self.up_conv2 = up_conv(256, 128, 3, 1, 1, 2, 'instance', 'relu') self.up_conv3 = up_conv(128, 64, 3, 1, 1, 2, 'instance', 'relu') self.up_conv4 = up_conv(64, 32, 3, 1, 1, 2, 'instance', 'relu') self.up_conv5 = up_conv(32, 3, 3, 1, 1, 2, None, 'tanh') ``` Experiments -------------------------------------------------------------- | Training config | Generated samples at 6400 iters | |----------|---------| |Basic without diff aug| | |Basic with diff aug| | |Deluxe without diff aug| | |Deluxe with diff aug| | From the results generated it can be seen that with deluxe augmentation, its better than basic augmentation. And with differentiable augmentation, quality improves very much. Deluxe augmentation has random-crop, color-jitter and random horizontal flip in addition to the basic augmentations. Differentiable augmentation is applied to both real and fake images and this helps the generator to replicate the augmented images accurately. Hence, with deluxe and differentiable augmentation, we get the best quality of generated grumpy cat images.   If a GAN manages to train, the discriminator loss has to be high and generator loss should be very low. This implies that the discriminator cannot distinguish between real and fake images and the generator is able to generate images that are indistinguishable from real images. With training for more number of iterations we should be able to see this behaviour. We can almost see this behavior with deluxe and differentiable augmentation. | Image at 200 iters | Image at 3000 iters | Image at 6400 iters | |----------|---------|---------|---------| | | | | Shown above are the generated images while the GAN is training. We can see that first the overall structure and dominant colors of the grumpy cat are learnt and subsequently the finer details of the face like eyes, grumpy mouth, whiskers and the colour are learnt. The details and the quality becomes better as the iterations go by. CycleGAN ============================================================== 1000 iters -------------------------------------------------------------- | cycle consistency? | X->Y | Y->X | |----------|---------|---------| | without | | | | with | | | 10000 iters -------------------------------------------------------------- | cycle consistency? | X->Y | Y->X | |----------|---------|---------| | without | | | | with | | | | cycle consistency? | X->Y | Y->X | |----------|---------|---------| | without | | | | with | | | From the table above, it can be seen that with cycle consistency loss, we get better results. In the cat images, especially in the russian blue to grumpy, we can see that with cycle consistency loss we get better quality eyes reconstructed. In the fruit conversion results, we see that without the cycle consistency loss, the model is able to identify the region where the fruit is and domain translation has to occur but it's not able to keep the translation sharp. The color spills all over the place. This doesn't happen with cycle consistency loss and we get better images. This is expected because of the mechanism behind cycle consistency which is to translate the image from one domain to another and back to the original domain and that reconstructed image has to match the original image as much as possible. DCDiscriminator vs PatchDiscriminator -------------------------------------------------------------- | Discriminator | X->Y | Y->X | |----------|---------|---------| | DC | | | | Patch | | | | DC | | | | Patch | | | Seen above, with DCDiscriminator image translation is barely happening. PatchGAN discriminator gives better results because it captures both local features in the patch and global features while the DCDicriminator only captures the global features. Bells & Whistles ============================================================== Pre-trained diffusion model -------------------------------------------------------------- I used stabilityai/stable-diffusion 2.1 model and gave the prompt 'grumpy cat coding infront of a computer'. Here are the generated images -     Spectral Normalization -------------------------------------------------------------- Spectral normalization helps in bringing in stability in the GAN training. I applied spectral norm to DCGAN with basic differentiable augmentation. We can see below that the loss graphs are more stable and the results are better (finer details like in the eye and the colors) with spectral normalization applied.   | without spec norm | with spec norm | |----------|---------| | | |