16726-Learning Based Image Synthesis-Spring 24

Assignment #3 - Cats Generator Playground

Student Name: Peipei Zhong, Andrew ID: peipeiz

Part 1: Deep Convolutional GAN

1，calculate the padding size

$output\_size = [\frac{input\_size - kernel\_size + 2\times padding\_size}{S}] + 1$ , [] means take the integer part of it.

output_size = (2 * output_size - 4 + 2 * padding_size) / 2 +1,

so padding should be 1 in this case.

2, Generator and discriminator training loss

1) --data_preprocess=basic, without differentiable augmentation

2) --data_preprocess=basic, with differentiable augmentation

3) --data_preprocess=deluxe, without differentiable augmentation

4) --data_preprocess=deluxe, with differentiable augmentation

Briefly explain what the curves should look like if GAN manages to train.

Discriminator Loss: at the begining, discriminator loss may decrease as the discriminator gets better at distinguishing real images from fake ones, or it may temporarily near 0 when D outperforms G. Then, as the generator imporves, it would oscillate. And then it would stabilize around 0.5 that indicates it is unsure whehter the inputs are real or fake, and this is what we want——this scenario suggests that G has become good at generating images that are very similar to real ones, making it challenging for D to distinguish between them. Generally we don't want D loss to converge to 0, because this means D is very confident in its predictions and G is not performing well enough to fool D, and the adversarial training might not lead to the desired improvement in G.
Generator Loss: at the begining, generator loss would be high,and gradually decreases as the generator learns to produce more convincing images. And it may fluctuate, due to the adversarial nature of the training process. Finally, it would stabilize within a range, which means the training process is reaching an equilibrium.

Samples

Discussion

The use of deluxe and differentiable augmentations significantly improves the quality of the cat images output.
This two methods improves the quality by sharpening the edge and adding more details, and making the output looking more realistic.
The quality also improves as the iteration increases.

Part 2: CycleGAN

1, without the cycle-consistency loss, train_iters = 1000

with the cycle-consistency loss, train_iters = 1000

2, without the cycle-consistency loss, train_iters = 10000

with the cycle-consistency loss, train_iters = 10000

3, Apple2orange

without the cycle-consistency loss, train_iters = 10000

with the cycle-consistency loss, train_iters = 10000

4, Difference between the results with and without cycle consistency loss:

With cycle consistency loss, it can generate better images that have more details, like sharper edges. This is because cycle consistency loss enable the model to maintain the detail in the translated images so that it could be translate back to the original ones.
But from the naked eye, I think the effect without using cycle consistency loss is better. There are different types of transitions from apples to oranges/ oranges to apples in the output images. When using cycle consistency loss, it seems that only the color has changed, the shape and texture of the whole have not changed much.

5, DCDiscriminator vs PatchDiscriminator

result for DCDiscriminator, itera = 10000

(Though the output images looks terrible, the loss curve is somehow resonable.) Loss curve for orange2apple

Compared to PatchDiscriminator, DCDiscriminator have much worse outputs. The result of DCDiscriminator is more blurred and with artifact.

This is because DCDiscriminator consider the images as a whole, so as long as the overall images looks similar to the real image, the generator won't be punished. While PatchDiscriminator focus on local areas or "blocks" of the image. It independently evaluates whether each small piece is real and averages these judgments to obtain the final output. This method makes the generated images more refined and realistic in texture and detail, catching more details.

	basic	basic+diff_aug	deluxe	deluxe+diff_aug
iter = 400
iter = 5800