Overview
This assignment focuses on neural style transfer where we transform a content image to fit a certain artistic style. The transformation happens by optimizing pixels in the input image with respect to content and style loss. These losses are defined in the feature space of a pretrained neural network and we use VGG19 for this purpose. Details of each loss are described in their respective sections.
Part 1: Content Reconstruction
Content Loss is a metric function that measures the content distance between two images at a specific depth of the neural network. The loss is simply a squared L2-distance of the features between the two sets of images (input and target) at layer \( L \). \[ | f_{X}^{L} - f_{c}^{L} |_{2}^{2} \]
Experiment : Content loss at different layers
The effect of optimizing content loss at different layers can be seen below
Based on the above results, adding content loss after "conv_2" and "conv_4" layers seems to work the best. For the next experiment, content loss is added after "conv_2" layer.
Experiment : Reconstruction from noise based on content loss
We start with two noise images sampled from a normal distribution and optimize them to fit the content of the original image.
With content loss after "conv_2" layer, the reconstructed images looks exactly like the content image, without any noise artifacts.
Part 2: Texture Synthesis
Texture synthesis relies on style loss which is a measure of the distance between the styles of two images. For the purpose of this assignment Gram matrix is used as a measure of style. It is the correlation between the channel dimension for each pixel (H, W) location. For the \( L^{th} \) layer with feature image \( f_k^{L} \), \[ G = f_k^{L}. (f_k^{L})^{T} \]
Experiment : Style/Texture loss at different layers
The effect of optimizing style loss at different layers can be seen below
Based on the above results, the configuration where style loss is added after each of "conv_1, conv_2, conv_3, conv_4, conv_5" layers and summed performs the best. The performance is similar with "conv_1, conv_2, conv_4, conv_8, conv_15" configuration but evaluation at increased network depth increases compute time.
Experiment : Reconstruction from noise based on style loss
We start with two noise images sampled from a normal distribution and optimize them to fit the given style image using the "conv_1, conv_2, conv_3, conv_4, conv_5" configuration.
Starting from different noise values we observe different outputs for optimized styles. The generated styles look thematically similar but are different at a pixel level.
Part 3: Style Transfer
A complete style transfer is achieved by inserting both the content and style loss in the network and jointly optimizing over them.
Experiment : Hyperparameter tuning
For this experiment, the weights for content loss and style loss are ablated to see the effect on the final output. Also, the position of content loss is varied to control the blend of style and content in the final output.
| Parameters/Images |
|
|
|---|---|---|
| content_weight=\(1\) style_weight=\(10^3\) |
|
|
| content_weight=\(1\) style_weight=\(10^6\) |
|
|
| content_weight=\(1\) style_weight=\(10^8\) |
|
|
Based on this experiment, \( \lambda_{content} = 1\) and \( \lambda_{style} = 10^6\), along with content loss after "conv_4" seem to be the best hyperparameters.
Experiment: Style Transfer results
The results shown below are generated by optimizing content loss after "conv_4" and style loss after "conv_1, conv_2, conv_3, conv_4, conv_5" layers. Also, the input image is initialized from content image instead of random noise.
| Style / Content |
|
|
|
|---|---|---|---|
|
|
|
|
|
|
|
|
Experiment: Style Transfer with different initializations
The output of style transfer is sensitive to how the input image is initialized. A comparison between random noise initialization and content image initialization is shown below.
--
--
Runtime: 18.22 secs
Runtime: 12.61 secs
Qualitatively, the output with content intitalization looks better than that from random initialization. For content initialization the structure of the content image is better preserved in the output and the style is not overly dominant. In terms of convergence, content initialization converges faster than random initialization as seen from the runtimes.
Experiment: Style Transfer on favorite images
The results shown below are generated using the same parameters as the previous section.
Bells and Whistles
Applying style transfer on outputs of previous homeworks (poisson blending)
