I. Introduction

For this assignment, we experiment with neural style transfer by inserting content-space loss for content reconstruction and style-space loss for texture synthesis to different layers of the feature extractor of a pre-trained VGG-19 network. We first start with implementing the 2 losses, seeing how inserting them at different layers in feature space affects the generated image. We then combine them and perform neural style transfer on an input image given a style image and a content image. For bells and whistles, I stylized grumpy cats.

II. Content Reconstruction

The content loss is the squared L2 distance of the feature of input image and the target content image at a certain layer. I get the feature extractor from a pre-trained VGG-19 network from torchvision.models, fix the network weights, and optimize the input image with LBFGS quasi-newton optimizer.

Below I experimented with adding the content loss after different layers and run optimization for 1000 steps.

conv_1

conv_2

conv_4

conv_6

conv_10

conv_2, conv_4

We can see that the content loss works better in earlier layers (conv_1, conv_2) while the the output image contains more noise as the layer goes deeper (conv_6, conv_10). This is because the initial layers of the network capture low-level features such as edges and textures which are closer to the raw pixel values. Optimizing content loss in these layers will result in reconstructions that closely resemble the original image. The deeper layers lack spatial details while capture more high-level features, therefore optimizing content with these abstractions will introduce artifacts and noise.

I chose conv_2 for the layer to add content loss and then experiment with 2 random noises as 2 inputs.

Visually there is no difference in the output image with different random noise. Quantatively the distance between the 2 images is also very close to 0. The reconstructed image does appear more blurry than the original image though, which is expected as high-frequency details are lost after the pooling layers in VGG-19 which reduce the spatial dimensions of features maps as the image passes through the network. By optimizing content reconstruction at conv_2, it natually focuses on minimizing large-scale differences while not perfectly capturing the precise alignment of every pixel.

III. Texture Synthesis

We can measure the distance of the style of 2 images with the gram matrix, which gives the correlation of two vectors on every dimension and is calculated by mulitplying the feature map matrix with its transpose for every layer. The style loss is then the squared L2 distance of the gram matrix of the input image and the target style image at specified layers.

Below I experimented with optimizing style loss at different layers and run optimization with 1000 steps.

conv_1, 2, 3, 4, 5

conv_6, 7, 8, 9, 10

conv_11, 12, 13, 14, 15

conv_1, 2, 4, 5, 6

conv_1, 2, 5, 6, 7

conv_1, 3, 5, 7, 9

I chose to add style loss at layers conv_1, 2, 4, 5, 6 for the below images and the following experiment with 2 random noises as 2 input images.

We can see that there is some differences in the generated texture with different noise initializations but both capture the general style well.

IV. Style Transfer

We then combine the content loss and style loss to perform neural style transfer. I tune the hyper-parameters and normalized the gram matrix over the feature pixels by dividing the total number of elements in the feature map. Below is a experiment with different weights for the style loss.

style_weight = 5x10^5

style_weight = 10^6

style_weight = 20^6

style_weight = 10^6 gives a good balance between the content and style. For other hyper-parameters, I use content_weight = 1, num_steps = 500, content_layers =conv_2, style_layers = conv_1, conv_2, conv_4, conv_5, conv_6. This allow the generated image to capture the texture of the style image while preserving the content of the original image.

Below are some results of optimizing from 2 content images with 2 style images.

Then I experimented with taking random noise and content image as input to do style transfer.

The generated image from random noise initialization is dominant by the texture and can only vaguely sees the content of the original image. Meanwhile taking content image as input works well. The time for the optimization is shown in the caption. There is less than a second difference between the 2 images, which is expected as the optimization is done on the same input shape and number of steps.

Below I chose some images I like to do style transfer.

This result shows that neural style transfer works well on artistic style, but not as well on natural images. This ties to what discussed in class, that the generative models tend to work well with paintings and artistic images, and it is harder with realistic images. Below are some more images generated in Jason Anderson's style.

IV. Bells & Whistles

1. Stylize grump cat from homework 3

Below are some results of stylizing grumpy cat and russian blue cat with picasso and jason anderson style.