Learning-Based Image Synthesis

Assignment 4: Neural Style Transfer

Andrew Id: shubhikg

Task Description

In this assignment, we explore Neural Style Tranfer. Neural style transfer is a deep learning technique that merges the content of one image with the artistic style of another. It works by using convolutional neural networks to iteratively transform an input image to match the style of a reference image while preserving its content. This process involves defining a loss function that balances content and style reconstruction, allowing for the creation of visually appealing and artistic images. Neural style transfer has found applications in art generation, photo editing, showcasing its versatility and creative potential in the realm of artificial intelligence.

Part 1: Content Reconstruction

Content reconstruction in neural style transfer involves ensuring that the generated image preserves the key features and structures present in the original content image. This is typically achieved by minimizing the difference between the features extracted from the generated image and those extracted from the content image. The process usually involves passing both the content image and the generated image through a pre-trained convolutional neural network (CNN), like VGG, to extract feature maps at multiple layers. These feature maps capture the content information at different levels of abstraction.

After extracting the feature maps, a loss function is defined to measure the difference between the feature representations of the content image and the generated image. The commonly used loss function for content reconstruction is the mean squared error (MSE) or the L2 distance between the feature representations.

1.1 Optimizing content loss at different layers

The objective of this experiment is to investigate the impact of optimizing content loss at different layers of the VGG neural network architecture on the quality of the generated image. The experiment will provide insights into how optimizing content loss at different layers of the VGG network affects the performance of neural style transfer. It will help identify optimal configurations for achieving high-quality results. I experiemnted with conv layers 2, 4, 6, 8, 10 and 12. The results are as follows:

Conv 2	Conv 4	Conv 6
Conv 8	Conv 10	Conv 12

From the above results we can see that conv2 and conv4 layer results are good, while rest are detroiting in quality as the layer number increases. Hence for future experiments I plan to conv2 results.

1.2 Reconstruction result comparison starting from different noises

In this experiment, I started with 2 different random noises and tries doing content rescontruction using conv_2 layer. The results are:

Input Noise	Reconstructed Output	Content Image

From the above results we can see that even though the input noise is different in both of them, both the outputs are exactly similar to the content image. Hence both produce the same output.

Part 2: Texture Synthesis

Style loss is a crucial component in neural style transfer, representing the measure of the difference between the style of a reference image and the generated image. It is computed by comparing the Gram matrices of feature maps extracted from different layers of a convolutional neural network(i.e. VGG). The Gram matrix captures the correlations between the features, encoding information about the texture, patterns, and overall style of an image. By minimizing the style loss during optimization, neural style transfer algorithms can effectively transfer the artistic style of a reference image onto the input image, resulting in visually appealing stylized images.

2.1 Optimizing texture loss at different layers

The results for optimizing style transfer loss at different layers are:

Original Style Image	Conv 1, 2, 3, 4, 5	Conv 1, 3, 7, 5, 9	Conv 5, 7, 9, 11, 13	Conv 7, 9, 11, 13

From the above results, the best performance is with conv layers 1, 3, 5, 7, and 9. Hence I will use this for all future exepriemnts.

2.2 Reconstruction result comparison starting from different noises

In this experiment, I started with 2 different random noises and tries doing content rescontruction using conv layers 1, 3, 5, 7, and 9. The results are:

Input Noise	Reconstructed Output	Style Image

From the above results we can see that both the outputs look similar in style, however they are exactly same at every pixel. However, the style of both the images is very similar.

Part 3: Style Transfer

3.1 Tuning the hyper-parameters

In this part of the assignment, I tuned the hyper-parameter for style loss weight and trained to a lot of experiments to chose the best one. I varied on a log scale from 1 to 10^6. The results are:

The content image and the style image I used for this experiment are:

Content Image	Style Image

The results for hyper-parameter tuning experiments are:

Style weight = 1	Style weight = 100	Style weight = 10^4

Style weight = 10^5	Style weight = 5*10^5	Style weight = 10^6

From the above results, we can see that style weight of 10^5 produces the best results.

3.2 Style transfer results

Some style results are as follows:

Style/Content Image

3.3 Style transfer different initialization comparison

Content Image	Style Image	Random Init	Content Init

From the above results we can see that content initialization is able to produce better results than random initialization. We can see that with content initialization, the network can preserve the content of the content image better and content is more dominant than style in such case. In terms of runtime, content initialization converged at a faster rate than random initialization. This happened because the content of the image was already initiliazed, hence the network took less time to learn the style of the image.

3.4 Results on my own images

Some more results on my own images are:

Content Image	Style Image	Style Transfer result