**16-726
Neural Style Transfer** **Anish Jain (anishaja)** Overview =========================================================================== In this project, we implement the Neural Style Transfer algorithm proposed by Gatys et al. in the paper [A Neural Algorithm of Artistic Style](https://arxiv.org/abs/1508.06576). The algorithm uses a pre-trained Convolutional Neural Network (CNN) to extract features from the content and style images. The content image is used to extract the content features and the style image is used to extract the style features. The algorithm then generates a new image that combines the content features of the content image and the style features of the style image. The algorithm uses an optimization-based approach to generate the new image. The optimization objective is to minimize the content loss between the content features of the new image and the content features of the content image, and the style loss between the style features of the new image and the style features of the style image. The algorithm uses a pre-trained CNN to extract the content and style features. We use the VGG19 model pre-trained on the ImageNet dataset to extract the features. The pretrained VGG19 model has multiple convolutional layers. Below is the architecture of the VGG19 model.  Content Reconstruction =========================================================================== I optimize a random noise image to match the content features of the content image. The content features are extracted from the VGG19 model at the 4th convolutional layer. The content loss is computed as the mean squared error between the content features of the content image and the content features of the new image. The optimization objective is to minimize the content loss. Vary content loss layer --------------------------------------------------------------------------- I experimented with different layers to extract the content features.             The layers in conv5 block do not help much. Looking at the above results, I decided to use conv4_2 layer to extract the content features.    Texture synthesis =========================================================================== I optimize a random noise image to match the style features of the style image. The style features are extracted from the VGG19 model at multiple convolutional layers. The style loss is computed as the mean squared error between the Gram matrices of the style features of the style image and the style features of the new image. The optimization objective is to minimize the style loss. Vary style loss layer --------------------------------------------------------------------------- I experimented with different layers to extract the style features.        Looking at the above results, I decided to use conv2_1, conv3_1, and conv3_4 layers to extract the style features.    Style Transfer =========================================================================== Noise Initialization --------------------------------------------------------------------------- Using content layers and style layers as discussed in aboce sections, I set the content loss weight to 1 and tune the style loss weight.       Some more results with tuned hyperparameters. | | |  | |----------------------------------|-----------------------------------------------------------------|----------------------------------------------------------------------| | || | | | | | Content Initialization --------------------------------------------------------------------------- I use the content image as the initialization for the new image. | | |  | |----------------------------------|------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| | || | | | | | Results on my own images --------------------------------------------------------------------------- | | |  | |---------------------------------------------------------|-------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| | | |  | | | | | Bells and Whistles =========================================================================== Return of cats ---------------------------------------------------------------------------   Stylized videos --------------------------------------------------------------------------- I tried stylizing a video from the 3DOVS dataset.  If you are not able to view the video, you can view it [here](https://drive.google.com/file/d/1C0DOYx4eXbd5IxrIXOhReCPrTkS2v391/view?usp=sharing) Feedforward neural style transfer --------------------------------------------------------------------------- I implement a feedforward neural style transfer network using the architecture of CycleGAN used in last assignment. The network is trained on the cat datatset. The results are not as good as the optimization-based approach. | | | | |---------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | || | | || |