Overview

In this assignment, we implement neural style transfer by optimizing content and style losses on different layers of a pre-trained VGG-19 network.

Part 1: Content Reconstruction

Content loss on earlier layers can precisely reconstruct the content of the image, while content loss on middle layers can only reconstruct the rough structure of the image with a lot of noise. Content loss on later layers can hardly reconstruct any content of the image, except some edges. Generally, when the layer is deeper, there is more noise in the reconstructed image.


We show the results of optimizing with content loss from conv1 layer. The reconstructed images are very close to the original images except for slighly blurry details on some high frequency regions, like the slightly smeared background in the dancing image and the fur in the Wally image.

Part 2: Texture Synthesis

Style loss on the first two layers gives synthesized results that have similar color patterns but less texture patterns, e.g., the results on conv1 and conv2 are mostly random color patches. Style loss on the middle layers gives synthesized results that have both similar color and texture, like the leaf textures in the results on conv3 and conv4. Style loss on the later layers gives mostly noise in the synthesized results.


We apply style loss on the first four convolutional layers with equal weights. The synthesized textures retain both the color and texture patterns of the original images.

Part 3: Style Transfer

We apply the content loss on the conv3 layer and the style loss on the first four convolutional layers. The style loss weight is set to 100, 1000, or 5000. For the gram matrix normalization, we normalize the gram matrix by the number of elements in the feature map, i.e., H * W. We use the mean reduction for the MSE loss hence the loss is further normalized by the number of features. The results are optimized for 300 iterations, with a learning rate of 0.05, and L-BFGS with strong Wolfe line search.



The results using content images as initialization retain more details of the content images and also converge faster, with lower final total loss. The style using content initialization is also more stable and similar to the original style image, e.g., patches exist in the noise initialized results using the Escher sphere image as the style but not in the content initialized results.