The goal of this assignment is to combine the artistic style from one image with the content from another. Given a content image, style image and an input image (either random noise or a copy of the content), the input image is optimized to balance content and style.
Part 1: Content Reconstruction
First we tackle preserving the content from the original content image.
Content Loss
The loss for the content is as follows
where the first f term is Lth-feature of X, where x is the input image. The second is the same but for C, the content image.
Feature extractor and Loss Insertion
We don't want to be using just plain old pixel loss. To avoid our context loss being equivalent to pixel loss at the 0th layer, we will compute our context loss in a feature space.
We use VGG-19 to extract features, as it can be easily imported from torchvision.models. After desired convolution layers, we append our context loss to the model.
Optimization
We are optimizing the image, and keeping the network fixed. This is opposite of what we did last homework, where we tuned the network, passing in fixed images.
We are using the torch implementation of the Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS). It's a quasi-Newton method that aims to approximate the original BFGS algorithm using less memory.
Experiment
Optimizing Content Loss at Different Layers
Lets take a look at what our content reconstruction images look like when we put the content loss after different convolution layers. We will specifically analyze our method on the wally image.
The further into the network, higher conv number, the worse we do in reconstructing the original image. Up until conv_4 all the images look about the same from a far, but when you zoom in (upper right corner) you can really see how the quality degrades. Since conv_4 is the latest layer where the reconstruction still looks fine to the naked eye, we will place our context loss after that layer.
Two random noises
For all the tests in this section we initialize the input image as random noise and optimize it. This results in slightly different outputs. Let's look at that now with our loss placed after conv_4.
Part 2: Texture Synthesis
Now that the context aspect is done, we can focus on the texture synthesis.
Style Loss
Like content loss we will once again be using MSE for the loss, but we need a way to measure style. So, we cannot use f^L_k("k-th dimension of the Lth-layer feature of an image") directly.
We will instead use Gram matrices in our MSE. The Gram matrix is calculated as follows
We than use the gram matrix of the predicted feature and optimized feature in our MSE.
Applying Loss and Optimization
These steps are implemented in a similar way to Feature extractor and Loss Insertion and Optimization in Part 1.
Experiment
Optimizing Texture Loss at Different Layers
Let's take a look at how placing style loss after different convolution layers affects what we learn about the style. We will use Edvard Munch's "The Scream" as our style input and random noise as our input image.
The further we place the style loss the finer detail we learn, but it can get too fine as seen in conv_10. What we learn at this level is not visually appealing. If we add style loss after several layers we can learn coarse and finer details of the style that make the final result more appealing.
We will use the last combo of convolutional layers, conv_2,4,6,8,10. This seems to capture colourful strokes well, while not being too choppy or too saturated. Of course my selection is due to personal preference, there really is not a right or wrong answer.
Optimizing Texture Loss at Different Layers
For all the tests in this section we initialize the input image as random noise and optimize it. This results in slightly different outputs. Let's look at that now with our loss placed after conv_2, 4, 6, 8 & 10.
Part 3: Style Transfer
Now that we have implemented both the content and style we can combine both of these to perform style transfer.
Hyper-parameters
We place content loss after the forth convolution in VGG-19. Style loss is placed after the second, third, forth, sixth, eighth and tenth convolutions. All layers after the last style last are removed. The style weight is 100000 and the content weight is 1.
Results on Given Photos
Content Input VS Noise Input
As you can see above, optimizing the content image gives a more visually appealing realest than optimising the noise. Though it does take about 1.5 - 2.5 seconds more to run on these 512x512 images. It is easier to get an appealing result when we are optimizing an image that already contains the main focus of the image, than trying to "recreate" the content in a noisy image.
