Assignment 4: Neural Style Transfer | 16-726 Learning-Based Image Synthesis | prabhdes

Please check Bells and Whistles in the end

0. Introduction

In this assignment, we aim to implement a neural style transfer algorithm, as proposed by Gatys et al. in "A Neural Algorithm of Artistic Style," which synthesizes an image by combining the content of one image with the artistic style of another. The algorithm utilizes a pre-trained Convolutional Neural Network (CNN), specifically the VGG19 model trained on the ImageNet dataset, to extract content and style features from the respective images. Content loss is computed as the sum of squared differences between the spatial feature maps of the content image and the synthesized image. Style loss is determined using the Gram matrix of the feature maps. The optimization objective is to minimize both content and style losses, resulting in an image that retains the content of the target image while adopting the artistic style of the reference image.

1. Content Reconstruction

1.1 Effect of Different Layers

In the process of content reconstruction using the VGG19 model, a random noise image is optimized to match the content features of the target image, which are extracted from the convolutional layer. The content loss is calculated as the mean squared error between the content features of the target image and those of the optimized image. As the illustration demonstrates, reconstruction results become increasingly noisy with deeper layers, due to their abstract nature and reduced relevance to the image's content. In contrast, earlier layers are more closely related to the content and hence are more effective for content reconstruction, resulting in a minimization of content loss during optimization.

1.2 Optimized With Random Noise

I chose layer Conv4_4.

2. Texture Synthesis

2.1 Effect of Different Layers

In our approach, we optimize a random noise image to replicate the style features of the target style image, using the VGG19 model to extract these features from multiple convolutional layers. We observe that textures synthesized from earlier layers tend to closely resemble the content of the original image, while textures from later layers are more abstract, capturing high-level information. The style loss is calculated as the mean squared error between the Gram matrices of the style features of the style image and those of the synthesized image, with the optimization objective being to minimize this style loss.

2.2 Optimized With Random Noise

I chose Conv_2_1 and Conv_3_1 for texture synthesis

3. Style Transfer

3.1 Hyperparameter Tuning

From the above hyperparameters and some others I'll be sticking to the following hyperparameters: Style Weight: 100000 Content Weight: 1 Content Layer: Conv3_1 Style Layers: Conv_1_2, Conv2_1, Conv2_2, Conv3_1, Conv3_2

3.2 & 3.3 Results of Optimization with COntent Image and Noise

3.4 Style Transfer on my images

A. Bells & Whistles

A.1 Grump Cats Results

A.2 Tried Style Transfer on Video with temporal smoothness