Assignment 4 - Neural Style Transfer

• Sihan Liu

0. Project Overview

In this assignment, my task is to implement the neural style transfer algorithm that generates a stylized image based on a given content image, with a similar style with a user specific style image. The algorithm involves optimizing a random input or a copy of the content image to match the content input in content distance space and the style input in style distance space.

In the first part of the assignment, we will implement the content loss by optimizing a random noise in content space to achieve a given content image. The results of different layers will be shown to visualize the ability of reconstructing content for different conv layer of VGG19. In the second part of the assignment, we will dive into the texture synthesis only, by using the style loss to optimize the generated image. We will test on its ability to generate texture given a style image from random noise. Lastly, the neural style transfer is to combine the content reconstruction and the texture synthesis all together. So in the third part, I will show more examples of the results and do the ablation study of layer choosing. For the Bells & Whistles, I stylized the cute dogs and use a feedforward network to output style transfer results directly .

1. Content Reconstruction

The content loss is a metric function that measures the content distance between two images at a certain individual layer. In this task, I use VGG19 to extract features. We generate the reconstruction image from random noise with different seed, and optimize it in pixel space. For some specific layers, the feature of both input content image and the generated image are extracted and calculate the mse loss of two feautures. Then this loss is used to backpropogate through the network and optimize the reconstruction image. To implement this content loss, I added a content_loss layer after each convolutional layer.

To run content reconstruction:

python run.py images/style/frida_kahlo.jpeg images/content/fallingwater.png reconstruction

1.1 Effect of optimizing content loss at different layers

Visualization of the results supervised by different layers:

origin	conv1	conv2	conv3	conv4	conv5

Comments:

• The reconstructed results are highly related to the layer we supervise the reconstruction loss. An early layer (e.g. conv1) will reconstruct the content with more details, while a later layer (e.g. conv5) will reconstruct the content with more global information. To balance the details and global information, we choose to supervise with the reconstruction loss at conv4.

1.2 Visualization of the results with different input noises

I think conv4 preserves reasonable content information and kind of remove the original texture or styles of the image. So, here is the visualization of the results supervised by conv4 (my favorite) with different input noises:

origin	noise 1	noise 2

2. Texture Synthesis

In this part, we use VGG19 only for texture synthesis. The style image and the generated image both feed through VGG19 and for some specific layers, the extracted feature difference is calculated using Gram matrix. The Gram matrix is used as a measure of style because it represents the texture or patterns in an image, rather than the actual content of the image. It is a square matrix obtained by multiplying the matrix of features with its own transpose. The resulting matrix captures the statistical relationships between the different feature maps in the input image.

To run texture synthesis:

python run.py images/style/frida_kahlo.jpeg images/content/fallingwater.png texture

Effect of optimizing texture loss at different layers

Visualization of the results supervised by different combos of conv layers:

origin	(1)	(1, 2)	(1, 2, 3)	(1, 2, 3, 4)	(1, 2, 3, 4, 5)

An early layer (e.g. conv1) tend to synthesize the texture with colors more similar to the source image. The generated texture is very blury and doesn't represent the shape and stroke of the original image very well and it collect the low peak information. However, when the layer is too deep, it collect some high peak information but the noise level is very high and the texture doesn't preserve. To combine both local and global style of the source image, we set the texture loss at conv1, conv2, conv3, conv4, conv5 by default.

Visualization of the results with different input noises

Visualization of the results supervised by conv4 (my favorite) with different input noises, because I think conv4 can best extract both textures and high-level details:

origin	noise 1	noise 2

3. Style Transfer

To implement style transfer:

python run.py images/style/frida_kahlo.jpeg images/content/fallingwater.png transfer

3.1 Tune hyper-parameters

In the Gram matrix, I normalized the matrix based on the number of pixels and I tune the hyper parameters to get the best result. We use the falling water image and stylize with the frida kahlo image. I apply the content loss on conv_4 layer and the style loss on conv_1, conv_2, conv_3, conv_4, conv_5 layers. Also, the content_weight is 1.0 and we take 300 optimization steps.

Style Weight	origin	1e3	1e4	1e5	1e6	1e7 (Better)
Result

3.2 Grid of Results

3.3 Comparison of the results with input

	content	style	input noise	input image
result
running time	N/A	N/A	17.76 s	16.87 s

Comments:

• The result from noise can express the style of the style image and visualize it in a gaussian distribution to some extent. However, the result from input image is more natural and has more details from the content image.
• The running time on input noise is more time costing, which is reasonable since we need to optimize the input noise from a more general distribution, that is a gaussian distribution.

3.4 Visualization of my favorite results

	content	style	output
transfer 1
transfer 2

4. Bells & Whistles:

Stylize cute dogs (2pts)

	content	style	output
transfer 1
transfer 2
transfer 3

Use a feedforward network to output style transfer results directly (8pts)

I use a feedforward network to output style transfer results directly. Run my code by

python style.py train --style-image style_imgs/mosaic.jpg

transfer	content	style	output
transfer 1
transfer 2
transfer 3