Neural Style Transfer

Brief introduction

In this homework, I implemented neural style transfer. The project could be divided into three parts. First is implementing content losses, second is style loss, followed by a combination of the two. The overall aim is to generate an image that maintains the content of one image but of the style specified by the input.

Content loss

The content loss is calculated by the mean squared error between the feature maps of the generated image and the target image. The feature maps are extracted from the VGG19 model. With pretrained VGG19 from torchvision, we could extract feature maps from different convolutional layers. I tried content losses on different layers insides VGG19, compared the reconstructed image with the target image, and picked one layer to use for the content loss. The MSE loss is calculated as follows: \[ \text{Content Loss} = \frac{1}{n} \sum_{i=1}^{n} (x_i - y_i)^2 \] where \(x_i\) is the feature map of the generated image, and \(y_i\) is the feature map of the target image.

Experiment 1: content loss on different layers

I tried content losses on different layers in this section. My comparision is between the following layers:

• conv_1
• conv_2
• conv_3
• conv_4
• conv_5
• conv_6
• conv_7
• conv_9

where postfixes denote indexes of convolutional layers in VGG19, no matter which block it is.

Here are the results from dancing.jpg and phipps.jpeg. Each image started with a random noise, and iteration number is fixed to 300.

Overall, conv 1 to 3 showed best results. This might because lower layers capture more detailed features such as shape and color, while higher layers capture more abstract features. The results from conv 4 to 9 are not as good as the ones from conv 1 to 3. They are either darker than original images or have more noise. I would pick conv 1 to conduct the following experiments considering efficiency.

Experiment 2: reconstruction result on two noises

I tried content loss on two noises to test stability of my implementation.

The differences between these reconstruction results and original images are so trivial that I could hardly observe anything.

The differences are also trivial.

Texture synthesis

In this section, I did experiments related to style loss. The idea is similar, we are computing MSE losses from two images, the output and the original style image. The special thing here is that we are computing the Gram matrix of the feature maps. The Gram matrix is calculated as follows: \[ \text{Gram} = F^T F \] where \(F\) is the feature map of the image.

Experiment 1: style loss on different layers

I picked out some combinations below. The total style loss is the summation of all style loss layers. Besides, since we don't need to add content loss in this part, I set the weight of style loss to 1 to avoid too large losses and gradients.

• set1: conv_1, conv_2, conv_3, conv_4, conv_5
• set2: conv_6, conv_7, conv_8, conv_9, conv_10
• set3: conv_11, conv_12, conv_13, conv_14, conv_15
• set4: conv_1, conv_3, conv_5, conv_7, conv_9
• set5: conv_2, conv_4, conv_6, conv_8, conv_10

Personally, I think set1 and set5 performed well in synthesizing the texture. I would prefer set1 more, because it preserved swirls and color in starry night while set5 has a lot of dark red components. So I would use set1: conv_1, conv_2, conv_3, conv_4, conv_5 for the following experiments.

Experiment 2: synthesis result on two noises

Below shows the synthesized textures from two noises. We can see that both maintains the geometry shapes and color from the original picture.

Style transfer

In this section, I combined content loss and style loss to generate images that maintain the content of one image but of the style specified by the input. The total loss is the summation of content loss and style loss. I fix the weight of content loss to 0.001 and adjusted the weight of style loss. This is becuase style losses tend to be a large number, which could easily cause gradient explosion. Since the ratio of content loss and style loss is vital, I divided them by 1000 to make the numerical values steadier.

Experiment 1: style weight

Note: When deciding hyper parameters, I used noises only.

As far as I can see, the second row is brighter and better balances the content and style. The first row resulted from small style and content weights, making it think the optimized result is good enough under a small loss.

It could also be observed from the second line: when the weight grows, there are more yellow chunks in the picture. For me, when weight = 10, it looks more natural. I would love to pick style weight = 10 together with content weight = 0.001 for the following experiments.

Experiment 2: Let's try!

Now we could try the neural style transfer on datasets. The style images are interpolated to the size of content images.