**Neural Style Transfer** # Introduction In this assignment, we implement style transfer methods to change a particular content image to resemble a certain artistic style. The methods are then tested on multiple images from the assignment, ,my own images, a video and some grumpy cats! # Content Reconstruction First we try to re-create the content image from noise using L-BFGS optimizer. We just use the content loss in this section ## Effect of optimizing content loss at different layers We are using VGG-19 network to extract features from the content (and style) image. VGG-19 has 16 convolutional layers. To reconstruct the content properly from just noise, and later use it for style transfer, we need to figure out which conv layer feature we can use. Earlier layers have higher spatial dimension and very less higher dimensional features. So though they might lead to a better reconstruction from noise, it might hinder the style transfer process as the algorithm, might try to preserve the structure of the content too much. Moreover the higher spatial dimension can slow down the process of calculating the content loss. With this in mind shown below are some reconstructions from noise using features from various conv layers at different depths for the content loss. All results were obtained with 300 iterations.

We can see that as we take features from deeper conv layers, the quality of the reconstruction generally drops. It especially visible in the noisy reconstruction based on the `conv8` features. The table below shows the mean absolute error (MAE) of the reconstructions: | Feature Layer | MAE | |-------|-------| | Conv1 | 0.0027 | | Conv2 | 0.0094 | | Conv3 | 0.0069 | | Conv4 | 0.1389 | | Conv5 | 0.2114 | | Conv8 | 0.2388 | From these experiments, it seems `Conv1` to `Conv4` is a good choice for calculating the content loss. We choose `Conv3` to get our future results. ## Content Reconstruction from Random Noise To test the robustness of the optimization we try to reconstruct the content image from two different noise initializations. The results are shown below. We can see from the images and the MAE values, that changing the noise does not significantly change the reconstruction. In fact they are indistinguishable to the naked eye. # Texture Synthesis In this section, we try to see how we can get aesthetic textures from the style image. ## Effect of optimizing style loss with different layer combinations On experimenting with feature combinations from different layers we see the following outputs. The caption for each image depicts the conv layers whose features were combined in the gram matrix to get the reconstructed texture.

On inspecting all the results subjectively it seems combining the earlier layers gives a more aesthetic and representative texture. However combining too few layers somewhat retains the structure of the style image (like the yellow blobs), which I personally find undesirable. So I finally use `[Conv1, Conv2, Conv3, Conv4, Conv5]` to calculate the style loss, as it represents the brush strokes quite accurately while retaining all the main colors of the style. Using deeper layers leads to grayer colors which is undesirable. ## Texture Synthesis from Random Noise We test the texture synthesis optimization from two different noise initializations. The results are shown below. At first glance the texture looks the same as they have the same pattern - which means our optimization is working well and giving a consistent texture for different initializations. On closer inspection subtle variations in the texture can be seen. # Style Transfer In this section we combine our findings from the last two sections and use it to apply a texture from the style image to the content image. ## Hyperparameter tuning and implementation details The Gram matrix was normalized by the number elements in each feature map. So a feature map of shape `a x b x c x d`, where `a` is the batch dimension and `b` is the number of feature maps, the Gram matrix was divided by `N = c x d`. The content loss was implemented using a L2 loss, and the style loss was implemented as a L2 loss on the normalized Gram matrices. The optimization was run for a maximum of 300 iterations. To vary the amount of style transfer, the content weight was kept fixed to 1 and style weight was varied relative to it. Shown below are results different style weights. In all the reconstructions below, the initializations was random noise. ## Some results on a 3x3 grid Results on two images and two styles are shown in the 3x3 table below. The style varies across the row and the content varies across the column. In all the style transfers below, the initializations was random noise. ## Initialization from Noise vs Content

Overall, though the time taken by both the methods is almost the same, the initialization from the content gives a subjectively better result with the main content of the image standing out, and the style being elegantly integrated into the image. ## Results on my images

# Bells and Whistles ## Style Transfer on a video ## Style transfer on grumpy cats