Assignment 4: Neural Style Transfer

Grace Su (Andrew ID: gdsu)

0. Project Description

In this assignment, I implemented neural style transfer. First, I implemented content loss to optimize random noise to match the content of a content image. Then, I implemented style loss to optimize random noise to synthesize textures that match the style of a style image. Finally, I combined the content and style losses to create a neural style transfer algorithm that can optimize a content image to match the style of a style image. I also implemented some Bells and Whistles: stylizing grumpy cats and applying style transfer for video (included at the bottom of this page).

1. Content Reconstruction

Implementation

I implemented content reconstruction by optimizing random noise to match the content of a content image. I used the VGG19 model to extract features from the content image and the generated image. I then computed the content loss as the mean squared error between the features of the content image and the generated image. I used the LBFGS optimizer to minimize the content loss and optimize the generated image to match the content of the content image.

1.1: Effect of Layer Choice on Content Reconstruction

I experimented with applying the content loss at different layers of the VGG19 model to see how it affects the content reconstruction results. The results are shown below. In general, the content reconstruction results are better when the content loss is applied at earlier layers of the model because the features extracted at earlier layers retain more low-level details of the content image.

The best results were obtained when the content loss was applied at conv_1.

1.2: Effect of Random Noise on Content Reconstruction

My favorite content reconstruction results were generated when the content loss was applied at conv_1. Next, I experimented with different initializations of the generated image to see how it affects the content reconstruction results. The results are shown below. There are no visible differences between the different random noise initialization.

2. Texture Synthesis

Implementation

I implemented texture synthesis by optimizing random noise to synthesize textures that match the style of a style image. I used the VGG19 model to extract features from the style image and the generated image. I then computed the style loss as the mean squared error between the Gram matrices of the features of the style image and the generated image. I used the LBFGS optimizer to minimize the style loss and optimize the generated image to synthesize textures that match the style of the style image.

2.1: Effect of Layer Choice on Texture Synthesis

I experimented with applying the style loss at different layers of the VGG19 model to see how it affects the texture synthesis results. The results are shown below. In general, the texture synthesis results seem to be better when the style loss is applied at earlier layers of the model.

The best results were obtained when the style loss was applied at [conv_1, conv_2, conv_3, conv_4, conv_5].

`[conv_6, conv_7, conv_8, conv_9, conv_10]`

`[conv_1, conv_3, conv_5, conv_7, conv_9]`

2.2: Effect of Random Noise on Texture Synthesis

Next, I experimented with different initializations of the generated image to see how it affects the texture synthesis results. The results are shown below. There are visible variations in generated textures between the different random noise initialization, but both results are plausible texture synthesis results.

3. Neural Style Transfer

Implementation

To implement neural style transfer, I used both the content and style losses to guide the optimizization of an initial image to match the content of a content image and the style of a style image. For the Gram matrix used in style loss, I made sure to divide by the number of elements in the feature map to normalize the loss. I also made sure to clamp the pixel values of the generated image to be between 0 and 1 after each optimization step. Additionally, I modified the dataloader so that it resizes and center crops the images to 512x512 to ensure the content and style images are always the same size.

3.1 Hyper-parameter Tuning

For the following experiments, I kept the style loss configuration at [conv_1, conv_2, conv_3, conv_4, conv_5].

I experimented with applying the content loss at conv_1 vs. conv_4 to see how it affects overall the style transfer results when initializing with the content image. The results are shown below.

The best results were obtained when the content loss was applied at conv_4 because it provided a good balance between content and style. I kept the content loss at conv_4 for the rest of the experiments.

I experimented with style weights of $10^5, 10^6, 10^7$ while keeping the content weight at 1 and initialzing with the content image. The results are shown below.

The best results were obtained when the style weight was set to 10^6 because it provided a good balance between content and style. I kept the style weight at $10^6$ for the rest of the experiments.

I experimented with content weights of $0.5, 1, 2$ while keeping the style weight at $10^6$ and initializing with the content image. The results are shown below.

The best results were obtained when the content weight was set to 1. I kept the content weight at 1 for the rest of the experiments.

3.2 Style Transfer Results

3.3 Initialization: Random Noise vs. Content Image

I compared the results of initializing the generated image with random noise vs. the content image. The quality of the results is better when initializing with the content image because the generated image already has the structure of the content image. The running time is faster when initializing with random noise (6.098801851272583 seconds) compared to the content image (6.125995635986328 seconds).