Assignment 4

16-726 | Qin Han | qinh@andrew.cmu.edu

1. Introduction

The goal of this task is to develop a neural style transfer algorithm that recreates specific content in a distinct artistic style. Content loss is calculated using the sum of squared differences between spatial feature maps in a Convolutional Neural Network (CNN). Style loss is determined by the Gram matrix of the feature maps in a CNN. Initially, we reconstruct the image focusing on content loss. Then, we synthesize the style based on style loss. In the final phase, we combine both content and style losses to achieve neural style transfer.

2. Content Reconstruction

2.1 Effect of Different Layers

The illustration below demonstrates that as we go deeper into the layers, the reconstruction results become increasingly noisy. This noise is attributed to the deeper layers' more abstract nature, making them less relevant to the image's content. Conversely, the initial layers are more closely related to the content. Therefore, these earlier layers are better suited for content reconstruction.

Content image	Conv 2	Conv 4	Conv 6	Conv 8

2.2 Effect of Different Noise

I choose layer conv_4. By starting from two different noises to reconstruct the image, the reconstructed images are quite similar to the content image.

Content image	Noise 1	Noise 2

3. Texture Synthesis

3.1 Effect of Different Layers

As shown below, the texture synthesized from the earlier layers are more related to the content of the original image, while the texture synthesized from the later layers are more abstract and capture the high-level information.

Style image	Conv 1	Conv 2	Conv 3	Conv 4	Conv 5

3.2 Effect of Different Noise

I choose conv_1 to conv_5 for texture synthesis. By starting from two different noises to synthesize the texture, the synthesized textures are similar to the style image.

Style image	Noise 1	Noise 2

4. Style Transfer

4.1 Transfer Results

By combining the content loss and style loss, the style transfer algorithm is able to generate images that preserve the content of the content image while adopting the style of the style image. In detail, I set the style loss weight as 10, and the content loss weight as 1, and optimize the image for 300 iterations. Also, I use conv_4 for content loss, and use conv_1 to conv_5 for style loss.

4.2 Content & Noise

The results below demonstrate that using content image initialization better preserves the structural details of the content images. In terms of running time, content image initialization converges slightly faster than random noise initialization.

4.3 Custom Results

Content Image	Style Image	Output Image