Max Grebinskiy's Home Page for 16-726 Project 5

GAN Inversion

In this first section, we aim to solve an optimization problem that'll take in a latent code and reconstruct an image of our choice. We do this by considering some loss function L, a trained generator G, and a real image x, and aim to achieve

.
We experiment with adjusting the weight of the perceptual loss over a variety of images, along with a fixed weight of 10 to an L1 loss to our output. Here is the image that we experiment on:

and below are the results on it:

Vanilla Gan, Perceptual Weight = 0.1, Latent = z

Vanilla Gan, Perceptual Weight = 0.01, Latent = z

Vanilla Gan, Perceptual Weight = 0.001, Latent = z

StyleGan, Perceptual Weight = 0.1, Latent = w

StyleGan, Perceptual Weight = 0.01, Latent = w

StyleGan, Perceptual Weight = 0.001, Latent = w

In these results, we see that the vanilla gan with a random noise vector as the latent space generally leads to blurry results, regardless of the weight to the perceptual loss. The same somewhat applies when the StyleGan is used with a w latent space, though a perceptual weight of 0.01 is more clear; however, the content of the cat that we are interested in is missing, as the eyes are evidently different. When we use the StyleGan with the w+ latent space, we see that for a perceptual weight of 0.01, we actually get the correct eyes. Although the entire image as a whole seems off due to too much cat fur extending towards the edges, the actual style of the cat we experiment on is captured here. However, we miss out on the natural background, and instead replace it with cat. Overall, it makes sense that the StyleGan performs better than the vanilla Gan architecture, as it is designed to perform style transfer on our input image more effectively. Each of these combinations only takes at most a few seconds to run, with the stylegan ones taking longer compared to vanilla gan.

Scribble to Image

In this section, we showcase results that convert a sketch of a cat into an image. Although these outputs do resemble cats, there is a myriad of issues; the images are very translucent / excessively light, and they are generally blurry as a result. This is likely due to the lack of quality information in the sketch, and the numerous empty patches inside the sketch.