16-726 Learning Image Synthesis HW3¶
DCGAN Results¶
1. Basic Data Preprocessing, No Differentiable Augmentation¶
Setting: DCGAN, Basic Data Preprocessing, No Differentiable Augmentation:
We note that discriminator loss drops quickly, and stays low. This shows overfitting in discriminator, hence its always able to almost correctly classify real or fake images everytime. The generator is unable to fool the discriminator, and fails to learn a meaningful distribution.
Generator Loss
Discriminator Loss
Image at Iter 200
Image at Iter 6400
2. Basic Data Preprocessing, with Differentiable Augmentation¶
Setting: DCGAN, Basic Data Preprocessing, With Differentiable Augmentation:
Augmentation: Random Crop and Random Horizontal Flip on both real and fake images.
From discriminator loss curve, we note that it doesn't overfit and the loss keeps oscillating. This means that generator is able to fool the discriminator and learn some meaningful distribution of the images.
We acheive considerable improvement in generations with just differentiable augmentation. High fidelity generation, with fewer artifacts are making the cat look more realistic.
Generator Loss
Discriminator Loss
Image at Iter 200
Image at Iter 6400
3. Deluxe Data Preprocessing, No Differentiable Augmentation¶
Setting: DCGAN, Deluxe Data Preprocessing, No Differentiable Augmentation:
Augmentation: Random Crop and Random Horizontal Flip only on real images.
Here also, from discriminator loss curve, we note that it doesn't overfit and the loss keeps oscillating. However, the variance in the loss is less compared to [2]. The generations are not as great as the [2]. This trend shows that higher the variance in discriminator loss, better the results. This also shows differentiable augmentation is a better method compared to deluxe.
Generator Loss
Discriminator Loss
Image at Iter 200
Image at Iter 6400
4. Deluxe Data Preprocessing, with Differentiable Augmentation¶
Setting: DCGAN, Deluxe Data Preprocessing, With Differentiable Augmentation:
Generator Loss
Discriminator Loss
Image at Iter 200
Image at Iter 6400
CycleGAN Results¶
1. No Cycle-Consistency Loss¶
Russian Blue to Grumpy Cat: 1000 iterations.
Russian Blue to Grumpy Cat: 10000 iterations.
Grumpy to Russian Blue Cat: 1000 iterations.
Grumpy to Russian Blue Cat: 10000 iterations.
Apple to Orange: 10000 iterations.
Orange to Apple: 10000 iterations.
2. With Cycle-Consistency Loss¶
Russian Blue to Grumpy Cat: 1000 iterations.
Russian Blue to Grumpy Cat: 10000 iterations.
Grumpy to Russian Blue Cat: 1000 iterations.
Grumpy to Russian Blue Cat: 10000 iterations.
Apple to Orange: 10000 iterations.
Orange to Apple: 10000 iterations.
Conclusion:¶
Using cycle-consistency loss produce better quality generations. This is clear from the above experiment of translating from Russian Blue to Grumpy cat. With cycle-consistency loss, generations are meaningful cats while without cycle-consistency generations are distorted cats.
This result indeed corroborate the idea of cycle-consistency that if we modify loss to encourage consistent translation between different stages, it helps preserve the structure in the generations.
For the apples and oranges translations, we don't notice much difference in the performace as the structure of apples and oranges are more or less similar.
CycleGAN with DCDiscriminator¶
Russian Blue to Grumpy Cat : 1000 iterations.
Grumpy to Russian Blue Cat: 10000 iterations.
Oranges to Apples : 1000 iterations.
Apples to Oranges: 10000 iterations.
Conclusion:¶
DCDiscriminator shows a degredation in performance compared to Patch Discriminator, clearly visible from the cats example. The results are blurry and "jaggedy" in appearance. This can be attributed to the idea that Patch Discriminator focus on textures and local features, hence generator makes the local features of the object more consistent.
For apples and oranges, we don't notice much difference. It is because oranges and apples are comparitevly less texture than cats, hence with fewer features. To account for smooth features, we would need to model very small patches over high resolution images, to capture the finer details as well.
Bells and Whistles¶
Pre-trained Diffusion Model Generations¶
We setup Text to Image Stable Diffusion 2.1 and prompt it with "a funny russian cat"
A Funny Russian Cat
A Funny Russian Cat
Diffusion Model Implementation¶
- UNet implementation of 1, 2, 4, 8 dimension multipler for channels. Each dimension is set of 4 Conv2D+ReLU layers.
- Beta Scheduler
Results:
