Assignment #5 - Cats Photo Editing
Yu-Chen Lin
I. Description
In this homework, we try to do cat photo editing through the user input sketch or text prompt. The first part includes the operations on GAN, and the second part asks us to implement photo editing with a pre-trained diffusion model.
II. Inverting the Generator
Various Combinations of the Losses Including LP Loss and Preceptual Loss
I kept the weight of the L1 loss as 10 and only experimented on the perceptual loss. I tried five configurations for the perceptual loss including 1, 0.1, 0.01, 0.001, and 0.0001. The results are generated after 1000 iterations, and I used styleGAN with z latent space.
I think the effect of the weight of the perceptual loss depends on what we care about. For example, for the first image, the image generated by perc_wgt (0.01) is the only one that can reconstruct the green background, while the image produced by perc_wgt (1) can reproduce the eyes more successfully. For the second image, only the image generated by perc_wgt (1) can reproduce the palm. For the third image, every image seems to perform more consistently.
target
perc_wgt (1)
perc_wgt (0.1)
perc_wgt (0.01)
perc_wgt (0.001)
perc_wgt (0.0001)
Different Generative Models Including Vanilla GAN and StyleGAN
I tried both models using the z space with perc_wgt (0.01) and l1_wgt (10). The styleGAN produced better reconstruction. For example, we can observe the yellow eyes and the palm in the first two images. For the third image, the image generated from styleGAN seems to be larger than the target, while the one from vanilla GAN reconstructs the target better (though more blurry).
The reason that styleGAN performs better may be that it tries to condition the noise latent also on the feature spaces. On the other hand, vanilla GAN achieves the mapping in a more brute way because the noise is only presented as an input of the generator.
target
Vanilla GAN
StyleGAN
Different Latent Space (Latent Code in z Space, w Space, and w+ Space)
I used styleGAN with perc_wgt (0.01) and l1_wgt (10) on different latent spaces. For the first image, I think the z latent space performs the best as it can construct the yellow eyes successfully. For the second image, the z latent space also performs better as it can construct the palm. For the last image, the w and w+ latent space generates more convincing results. The w+ space is even better as the eyes it constructs are closer to the original target.
target
z
w
w+
target
z
w
w+
target
z
w
w+
III. Scribble to Image
Results
Both the w and w+ space plots perform well in this sparse sketch. The w+ plot is more consistent with the sketch.
target
z
w
w+
The dense sketch leads to more blurry generated output because the constraint may be too strong to generate details. Nevertheless, the w+ plot can generate quite beautiful eyes.
target
z
w
w+
This is also a dense sketch. The w+ plot seems to overfit on the sketch. The w plot produces more reasonable and realistic results though still a bit blurry.
target
z
w
w+
This sketch is more detailed and specified with more color. Although both the w and w+ plots are blurry, it is still nice to see the generation of cat whiskers.
target
z
w
w+
The sketch includes some dots around the mouth. However, neither results successfully illustrate this. The reason may be that it is too hard to find this kind of cat in the distribution that the generator learns.
target
z
w
w+
The sketch specifies some eyes, but as we can see, the algorithm struggles to generate the eyes. Still, it can generate a pink mouth appropriately.
target
z
w
w+
IV. Stable Diffusion
Amounts of Noises Added to the Input
I used a variant number of timesteps to experiment with the amounts of noise added to the input. In general, if the timestep is higher, it generally produces more diffusion-model-style output and includes the background.
The image from timestep 700 is quite good, and the image from timestep 900 seems to be more distant from the original sketch.
"Grumpy cat reimagined as a royal painting"
timestep (500)
timestep (700)
timestep (999)
We can observe the original text transforms and is closer to "Walter White" in the conditioned text. Further, just like the previous image, the image from timestep 700 is closer to the original sketch, while the image from timestep 999 manually decodes the original sketch to the real look of "Walter White."
"Walter White reimagined as a child's artwork"
timestep (500)
timestep (700)
timestep (999)
The image from timestep 999 is also distant from the original sketch. The image from timestep 500 generates some unnatural small figures. The image from timestep 700 generates an image that is close to the sketch and the prompt.
"Godfather reimagined as a cartoon style"
timestep (500)
timestep (700)
timestep (999)
Classifier-Free Guidance Strength Values
We know that when the strength value goes higher, the generated output will be more related to the conditioned text. We can also observe this kind of tendency here.
We can see the image with strength 30 is more refined and elegant than other images.
"Grumpy cat reimagined as a royal painting"
strength (5)
strength (15)
strength (30)
The image produced with strength 30 does not look like what a child draws because it is too detailed. Maybe it is conditioned on the "artwork" more than the "child".
"Walter White reimagined as a child's artwork"
strength (5)
strength (15)
strength (30)
The image generated by strength 30 essentially overfits the word cartoon, which does not look appropriately. The one produced in moderate strength (15) still looks better.
"Godfather reimagined as a cartoon style"
strength (5)
strength (15)
strength (30)
