Max Grebinskiy's Home Page for 16-726 Project 4

Content Reconstruction

In this section, we aim to implement content-space loss, and be able to optimize a random noise with respect solely to this content loss. There are two main sections: optimizing the content loss at different layers, and then seeing how two random noisy inputs compare after getting optimized. For the first section, we showcase the result of using different layers of the content loss with the dancer:

We see that placing the content loss earlier on in the layers has the best effect, as it is a perfect match. Even when placed at the 6th layer, we find that it is pretty good, though it does not converge as quickly in the same number of steps, as there are some patches that are pixelated (such as the hand and hair). By the 8th convolutional layer, we start to notice differences where the result is not perfectly recreated.

In the second section, we have the following results:

Difference between Real Image and Reconstructed 1

Difference between Real Image and Reconstructed 2

As we can see, regardless of the initial noise that we start with, the reconstructed image always converges to the original image. There is virtually no difference between either reconstructed image after optimizations, as can be seen by the image-differences.

Texture Synthesis

In this section, we implement the Gram Matrix and style loss. There are two main sections: the effect of optimizing texture loss at different layers, and taking two random input noises and optimizing them purely with style loss. For the first section, we showcase the following results with the picasso painting style:

The captions below each image denote which layers were used when optimizing texture loss. As we can see, the first two combinations of conv layers produced similar results in terms of color composition, though the actual thickness of the brush strokes in the texture are thicker in the first combination compared to the second one. The third combination is very different, where the colors are generally much lighter, but the style of the brush strokes is still similar. When comparing the loss of the second combination to the third combination, we have 0.0751 versus 0.2955, respectively.

For the second section, we showcase the following results with the scream painting style:

Style Transfer

In this section, we combine style loss and content loss in order to apply styles from one image onto another image while preserving the content of the image. In the first section, we tune the hyperparameters of our model in order to get the best model. For each image, we used the conv1 layer for content loss and conv1-5 layers for style loss. For the images in other sections, we experimented with using layers 1,3,5,7,9 for style loss, which seemed to provide better results in certain contexts.

Style Weight = 100000, Content Loss = 1.5

Style Weight = 100000, Content Loss = 2.5

Style Weight = 100000, Content Loss = 3.5

Style Weight = 1000000, Content Loss = 3.5

From a qualitative perspective, it seems that increasing the content loss does not increase the quality of the image, and the style of the picasso painting is very dominant over the content. Even with the total loss being higher, the rate at which the loss decreased was very slow over iterations, so it does not seem to change significantly with the same layers used. With the style loss weights set to 1000000, we see that the picasso style has a much higher weight towards the image. For the 4th image, it has a total loss of 4.3828, and for the 3rd image it is 3.2297.

In this second section, we mix together different content images and styles to produce different results. More specifically, we take the content images of Wally and falling water and mix it with the style images of "The Scream" and "Starry Nights". We showcase the results below, with the content being horizontal and the style being vertical:

In this third section, we compare taking in random noise as the input versus a content image, and use the same style image for both. We specifically use the dancer as the content image and the picasso painting as the style. Here are the results for this:

Note that for this experiment, I adjusted the content loss weight hyperparameter from 1.0 to 2.5. This was because previous attempts entirely lost the figure to the picasso painting. In general, the two verions perform very similarly. The image generated with noise had a total loss of 0.9235 and took 130.016 seconds to finish, whereas the content clone had a total loss of 0.9019 and took 134.564 seconds to finish. From a visual standpoint, it seems that the noise is qualitatively better, as the dancer seems more distinct from the background, particularly in the hand and the face.

Personal Images

Here, I applied neural style transfer to my dog and a fun image of a cow and dolphin jumping in the water together.