16726-Learning Based Image Synthesis-Spring 24

Assignment #4 - Neural Style Transfer

Student Name: Peipei Zhong, Andrew ID: peipeiz

Overview

In this assignment, I implemented a basic neural style transfer. In style transfer, we make an image have the style of another image while preserving the content of another image(in some case it's the content of itself.) First, I start from random noise and optimize it in content space, then I only optimize to generate textures, and finally do style transfer.

The key of this assignment is the loss:

• content loss:

  $L_{content} = \frac{1}{2} \sum (F_{ij}^l - P_{ij}^l)^2$, F: feature of content image; P: feature of generated image, l: layer

• style loss:

  $L_{style}^l = \frac{1}{4N_l^2M_l^2} \sum (G_{ij}^l - A_{ij}^l)^2$, N: style image, G: generated image, l: layer, N: number of feature maps, M: the size of each feature map at layer l.

What do content and style mean in 'style transfer'?

• content: "content" refers to the fundamental structure, subject matter, and overall composition of an image. It embodies the recognizable features, shapes, and objects that make up the image, independent of the artistic style or aesthetic treatment applied to these elements.
• style: "style" refers to the distinctive visual elements, techniques, or aesthetics that characterize a particular artwork, artist, or art movement. These elements can include but are not limited to colors, textures, brush strokes, lighting, and spatial relationships within the artwork. The style essentially encapsulates the artistic expression that is independent of the specific content or subject matter depicted in the artwork.

VGG19:

In style transfer, it is used as a feature extractor, without the classification layer.

Some results:

inputs：

style image	content image

results with content_layers_default = ['conv_4'], style_layers_default = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

Performing Image Reconstruction from white noise initialization

Performing Texture Synthesis from white noise initialization

Performing Style Transfer from moise initialization

Performing Style Transfer from content image initialization

Part 1: Content Reconstruction

1, Effect of optimizing content loss at different layers.

conv_1	conv_3	conv_5	conv_7

We can see that the reconstruction effect of shallow layers is better than that of deep layers. This is because the feature map of shallow layer would capture basic information such as edges, lines and textures in the image, while deeper layers would capture more advanced features and abstract concepts.

2, Take two random noises as two input images, optimize them only with content loss.

I choose layer_3.

seed = 34	seed = 4

We can see that the reconstructed results are stable with different noise.

Part 2: Texture Synthesis

1，Report the effect of optimizing texture loss at different layers. Use one of the configurations.

input:

starry_night	the_scream

conv_1,2,3,4,5	conv_1,3,5,7,9	conv_2,4,6,8,10	conv_6,7,8,9,10

I think conv_1,2,3,4,5 best reflects the style of the input style image. It contains most color, and brushstroke patterns.

Conversely, conv_6,7,8,9,10 is the worse. Deeper layers seem to encode abstract and high level feature, and color is lost and texture is blur.

conv_1,3,5,7,9 and conv_2,4,6,8,10 have similar results, while the contrast of conv_2,4,6,8,10 seems higher.

2，Take two random noises as two input images, optimize them only with style loss.

I choose conv_1,2,3,4,5

seed = 34	seed = 4

We can see that with different noise, the output can both reflectd the style in the input style images.

Part 3: Style Transfer

Style Transfer Implementation detail and configuration:

In this task, we optimized the image based on both the content loss and texture loss that has been implemented in the above sections.

Based on the above experiments, I use content layer = 'conv_3', style layer = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5'], style_weight=1e5 and content_weight=1.

Style Transfer results:

style\content	tubingen	phipps	dancing
the scream
starry night
frida_kahlo

3, Take input as random noise and a content image respectively. Compare their results in terms of quality and running time.

Style: frida_kahlo

from noise	from content

• quality: generate from content image is better. The results from noise is blurry and still with some noise. In contrast, the image initialized from the content image has better quality and preserves more details.
• running time: The training times of these two kinds of input are similar and close to each other.

4, Try style transfer on some of your favorite images

style	content	result
Anime-<Violet Evergarden>
Scenery-black&write