Assignment #4 - Neural Style Transfer

Effect of Optimizing Content Loss at Different Layers

In the following figures, I demonstrated the effect of optimizing the content loss at different layers. The captions show the specific layer for optimizing the content loss and also the reconstruction loss. The reconstruction works better when using the shallower layer, which is reasonable because the initial layers encode original pixel-wise information. For the deeper layer, we can only reconstruct the sketch of the original image because the deeper layers capture more high-level features of the input image.

conv_1 (0.000016)

conv_2 (0.004481)

conv_3 (0.018272)

conv_4 (0.060652)

conv_5 (0.080320)

conv_6 (0.092030)

conv_7 (0.106151)

conv_8 (0.118492)

conv_9 (0.150873)

conv_10 (0.175892)

Results

I have just visited Phipps Conservatory and Botanical Gardens recently. That's why I chose to reconstruct the image. We can see that even though the initialization is only a random noise, using only the content loss, we can reconstruct the image successfully with very low reconstruction loss (around 1e-2). Note that I chose to use the default conv_4 features for optimizing the image. Further, the two reconstructed results are consistently good with different noisy initializations, showing that the content loss optimization skills are not affected by the different random initializations.

noise

content image

conv_4 (0.061332)

noise

content image

conv_4 (0.061428)

Effect of Optimizing Texture Loss at Different Layers

I tried 8 configurations with different consecutive layers or skipped layers. We can observe when using consecutive layers (the first three configurations), taking the deeper layers will lead to a less desirable effect (looks close to random noise). In general, configurations containing the shallower layers like conv_1 or conv_3 result in a better effect in reproducing the texture. Thus, it is not surprising that the configuration ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5'] produces the best effect.

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

['conv_6', 'conv_7', 'conv_8', 'conv_9', 'conv_10']

['conv_11', 'conv_12', 'conv_13', 'conv_14', 'conv_15']

['conv_1', 'conv_3', 'conv_5', 'conv_7', 'conv_9']

['conv_7', 'conv_9', 'conv_11', 'conv_13', 'conv_15']

['conv_1', 'conv_4', 'conv_7', 'conv_10', 'conv_13']

['conv_4', 'conv_7', 'conv_10', 'conv_13', 'conv_16']

['conv_1', 'conv_5', 'conv_9', 'conv_13', 'conv_17']

Using the best configuration ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5'], I also demonstrated the texture of other images. The effect is also quite good.

style image

style image

style image

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

Results

For the results, I also used the best configuration ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']. Even though the random initialization is different, the algorithm can still return quite reasonable results with vivid texture. If we try to compare the two generated textures, we can observe there are still some differences in the local details, which demonstrate some diversity. Still, both generated images successfully reproduce the texture of the style image.

noise

style image

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

noise

style image

['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

Hyper-parameters Tuning

For the implementation of the gram matrix, I followed the instructions and did the normalization by dividing the final results by (batch_size * channel * height * width).

From the results of previous sections, I decided to keep the content layers as ['conv_4'] because although ['conv_1'] may reconstruct the images better, it tends to capture too many details, which may be too harsh for style transfer. For the style layers, I selected ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5'] as before.

For the hyper-parameter tuning, I kept content_weight as 1 and tried to modify style_weight and num_steps. It's easier to pick the hyper-parameters from the results. Let's look at the results from a full grid search of style_weight = [1e5, 1e6, 1e7] and num_steps = [300, 500, 700].

We can see that the images generated by style_weight = 1e5 / num_steps = 700 works the best. It not only maintains the structure of the content image but also adapts to the style of the target image.

style_weight = 1e5 / num_steps = 300

style_weight = 1e5 / num_steps = 500

style_weight = 1e5 / num_steps = 700

style_weight = 1e6 / num_steps = 300

style_weight = 1e6 / num_steps = 500

style_weight = 1e6 / num_steps = 700

style_weight = 1e7 / num_steps = 300

style_weight = 1e7 / num_steps = 500

style_weight = 1e7 / num_steps = 700

Results

For most of the images, I stick to the setting of style_weight = 1e5 / num_steps = 700 which is obtained from the previous section. For phipps with starry_night, tubingen with starry_night, and wally with frida_kahlo, I instead used style_weight = 1e5 / num_steps = 500 to produce more reasonable results.

content images ->

style images

|

V

Initialization from a Random Noise or a Content Image

From the quality aspect, I do not think there are many differences. That is, both results are quite good. The only thing I observed is that the image generated by initializing from the content image has some brighter parts for the wall of the building.

content image

style image

init from noise

init from content

For the running time, both run around 24 seconds for 700 steps on a GPU. We expect if we initialize from the content image, the content loss should converge faster. We can confirm this idea from the following figure. With the content image initialization, the initial content loss is already good. With only a random initialization, the content loss converges around 400-500 steps.

Style Transfer on My Favorite Images

content images ->

style images

|

V