16-726 Assignment 3
David Krajewski - dkrajews
Project Summary
DCGAN Results
Basic without DiffAug
Discriminator Loss
Generator Loss
Basic with DiffAug
Discriminator Loss
Generator Loss
Deluxe without DiffAug
Discriminator Loss
Generator Loss
Deluxe with DiffAug
Discriminator Loss
Generator Loss
A trained GAN should have curves that show that the loss goes down to a sufficiently small value and stays there. Sporadic changes in loss and fluctations show that the GAN might not have been able to fully learn.
Examples from Deluxe with DiffAug
From Iteration 200
From Iteration 6400
As you can see, the sample from the early iteration (iteration 200) begins to learn some of the colors associated with the cat, but it fails to really understand any structure. After 6500 iterations, we can see that the model more or less learned the colors and structures of the cat, but still fails to generate a meaningful background.
CycleGAN Results
No Cycle Consistency Loss, 1000 iterations
After around 1000 iterations, you can see that the model begins to understand some of the color and structure, but it still struggles with a lot of the image and background especially.
With Cycle Consistency Loss, 1000 iterations
Immediately by adding the cycle consistency loss, you can see that the model now has some structure constraints built into it. After only 1000 iterations, it learns the structure fairly well and starts understanding some of the color, though it has a long way to go.
Patch Discriminator, With Cycle Consistency Loss, 10000 iterations
The longer training time allows the model to generate higher fidelity results. It's clear that it learned the colors fairly well (fur color, eye color especially), but struggles a bit to capture the structure of the gray cat.
Patch Discriminator, Without Cycle Consistency Loss, 10000 iterations
Looking at the results, it seems like removing the cycle consistency loss makes the model lose some of the higher level structure. For example, in the image on the left, it seems like it understand's more or less the centers of the cats' faces, but everything around it seems to be blurry and unrealistic. However, it seems like the colors were still decently learned without cycle consistency loss.
DC Discriminator, With Cycle Consistency Loss, 10000 iterations
One of the big advantages of using in the patch discrimniator is that it gives the discriminator some spatial understanding of the image, instead of being pixel-wise consistent. This is fairly evident in this result, where it seems like at a global level the structure is preserved, but if we look closer, the cat's shape seems to be fairly inconsistent with the source images.
Apples/Oranges, Patch Discriminator, With Cycle Consistency Loss, 10000 iterations
Apples/Oranges, Patch Discriminator, Without Cycle Consistency Loss, 10000 iterations
Comparing these results to those with cycle consistency loss, I noticed a few things. The first is that it's farily surprising how well it's able to keep the structure of the original image without any sort of strict enforcement in that direction. However, it's fairly evident the shortcomings of not using cycle consistency loss. The colors seem to be very inconsistent and wrong. If you look at the image on the left (apples -> oranges), the model thinks that oranges are very dark, which makes it look very unrealistic. Again, cycle consistency loss enforces this sort of structure/color consistency that makes the model significantly better at learning the data.
Apples/Oranges, DC Discriminator, With Cycle Consistency Loss, 10000 iterations
It is very evident the effects of the patch discriminator in this example, in my opinion. At a high level, the patch discriminator is giving us some spatial understanding of neighboring pixels in the image, as opposed to a standard DC discriminator which deals at the individual pixel level. For many of the images in this example, the individual fruits might look somewhat reasonable, but the context that they're in / the rest of the image looks wholly separate from the source image.
Bells and Whistles
Diffusion
I wrote all the code for diffusion, but ran out of time to debug what was going wrong. I was running into an issue where I don't believe I was sampling / training in a consistent way, leading to results that were basically pure noise. I am including my code just in case, so hopefully I can get at least some partial credit (please LOL). Anyways, if while grading you notice anything immediately wrong with my code, I would appreciate some insight into why this didn't work. I will be continiung to work on it even after the deadline since I want to understand diffusion!