Part 1: Inverting the Generator
Various Losses
The following results are from stylegan model on z space with 1000 iterations
L1 only, 0.01
L1 only, 1
L1 only, 10
L2 only, 0.01
L2 only, 1
L2 only, 10
perc only, 0.01
perc only, 1
perc only, 10
L1(10)L2(1)Perc(1)
L1(10)L2(1)
L1(10)Perc(1)
L2(1)Perc(1)
Observations: L1, L2, Perceptual Losses are all effective. They are most effective with weights at different magnitutdes.
Best option: The image quality is best when combining all three losses. The details of the cats eye is the sharpest in this image and the style is consistent with the original image.
Efficiency: I didn't see much difference between experiments. 1000 iterations all took about 20s to run.
Different Models
Original
StyleGAN
Vanilla
Observations: The StyleGAN image is more detailed and has a higher resolution. The VanillaGAN image is is faster but more blurry. (The experiments are run with 1000 iterations on z space with all three losses)
Best option: StyleGAN (Better quality) / VanillaGAN (Faster)
Efficiency: The vanilla GAN(~10s) is 2 times faster that StyleGAN(~20s)
Different Latent Space
z
w
w+
Observations: The W+ space yields superior results,the details on the cat nose is much better. This is because w+ is an expanded latent space capable of encoding more information than the smaller W space. This allows for a better fit to image data. Both W and W+ spaces outperform the Z space, as they contain semantically meaningful priors, unlike the Z space, which consists of random noise.
Best option: w+ space
Efficiency: w+ is slightly longer (~1s more) but the difference is barely noticeable (~20s)
Part 2: Scribble to Image
The following results are from stylegan model on w+ space with 1000 iterations
Sketch
Mask
Result
Observations: Denser scribbles poses stronger constraints on the optimization process, but also lead to loss in details and made the result less realistic. Sparse scribbles are more flexible and can be used to guide the model to generate better results with more realistic details(see 1 & 2). Denser sketch also made the coloring of the result less realistic (see 6,7,8). This is probably the model can not find a good balance between texture and coloring, or the dataset is not diverse enough to capture the coloring of the sketch.
Part 3: Stable Diffusion
Different Inputs
Original Image
Generated
"Cute furry cat as a princess"
"A curious cat as a detective from 1900s"
Input A, variable strength
Original
10
1.1
15
5
30
Observations: The strength values control how well the output align with the prompt. We can see that when strength is very low, the output is burry and generic. When the strength is high, the output aligns well with the prompt but suffers from over saturation problem. I found value around 10 is a good balance between style and original content.
Grumpy cat, variable steps
Original
1000
800
700
Observations: As we increase the amount of noise by increasing the number of timesteps, our synthesized image becomes less faithful and more realistic. The 1000 timestep result looks like a real painting (with frames and correct posture). But the 700 and 800 step one have more details of the original cat image.