Assignment 4: Neural Style Transfer

Student name: Tianxiang Lin
Andrew ID: tianxian

Project Overview

This project aims at implementing neural style transfer which resembles specific content in a certain artistic style.

Content Reconstruction

Effect of optimizing content loss at different layers

Report the effect of optimizing content loss at different layers.

I experimented the content loss on conv_1 to conv_5 respectively. By applying content loss at different layers, the deeper layer the content loss is applied, the more noisy result we will get.

Dancing

Content	conv_1	conv_2	conv_3	conv_4	conv_5	conv_10

Tubingen

Content	conv_1	conv_2	conv_3	conv_4	conv_5	conv_10

Results from random noises

Choose your favorite one (specify it on the website). Take two random noises as two input images, optimize them only with content loss. Please include your results on the website and compare each other with the content image.

According to results above, I prefer the content loss performing on conv_1. Therefore, I optimized the two random noises with content loss of conv_1. The results are shown below. The resulting images are close to the content images.

Content	Noise1	Output1 (conv1)	Noise2	Output2 (conv1)

Texture Synthesis

Effect of optimizing texture loss at different layers

Report the effect of optimizing texture loss at different layers. Use one of the configurations; specify it in the website

The configurations are as follows:
config1: ['conv_1'],
config2: ['conv_1', 'conv_2'],
config3: ['conv_1', 'conv_2', 'conv_3'],
config4: ['conv_1', 'conv_2', 'conv_3', 'conv_4'],
config5: ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

From the results below, I prefer the style loss at layers ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5'] because it gave results whose style is much better and sharper.

Picasso

Style	config1	config2	config3	config4	config5

Results from random noises

Take two random noises as two input images, optimize them only with style loss. Please include your results on the website and compare these two synthesized textures.

I tested on two different style images with different random noises. The style loss is applied at layers ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']. Even though they are visually different by shape, the generated outputs have good, consistent and sharp style compared to their style images.

Content	Noise1	Output1 (config4)	Noise2	Output2 (config4)

Style Transfer

Tune the hyper-parameters until you are satisfied. Pay special attention to whether your gram matrix is normalized over feature pixels or not. It will result in different hyper-parameters by an order of 4-5. Please briefly describe your implementation details on the website.

In my implementation, the content loss is applied at the layer of conv_4 and the style loss is applied at ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']. The content loss weight is fixed to 1. I experimented with style loss weights of 1e4, 1e5, 1e6, 1e7. The results are shown below. From the observations, when the style loss weight is set to 1e4, the performance reaches its best.

Please report at least a 2x2 grid of results that are optimized from two content images mixing with two style images accordingly. (Remember to also include content and style images therefore the grid is actually 3x3)

Take input as random noise and a content image respectively. Compare their results in terms of quality and running time.

The results from content images and random noises are shown below. From the observations, results tend to be less similiar to the content image when style weights become smaller. Among four selected weights, 1e4 performs the best because it retain the details of the content image while transfer the style appropriately. The running time differs when training on different images. Overall, it takes more time for results from content images than random noise inputs.

Content	Style	1e4	1e5	1e6	1e7
N/A	N/A	20.408632516860962s	21.5148868560791s	21.425130605697632s	22.415777444839478s
N/A	N/A	23.427430152893066s	23.185594081878662s	15.109795093536377s	11.991308450698853s

Content	Style	1e4	1e5	1e6	1e7
N/A	N/A	19.766030073165894s	21.549476146697998s	21.442376375198364s	21.327046632766724s
N/A	N/A	23.76131796836853s	23.42719292640686s	23.142351388931274s	12.0399649143219s

Try style transfer on some of your favorite images.

I uses the CMU campus view picture from Hamerschlag Hall as the content image, and Winslow Homer and Monet's paintings as style images. The content loss is set to 1 and applied at conv_4. The style loss is set to 1e4 and applied at ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']. The results are shown below.

Content	Style	Output