Overview

In this assignment, we explore the concept of neural style transfer. This involves using a technique that mixes the content of one image with the style of another. To do this, we apply two main types of losses: one that focuses on recreating the content of the original image and another that captures the artistic style of the style image. These losses are applied at different levels within the VGG-19 network, a pre-trained model known for its ability to recognize various features in images. Our first step is to develop and understand these two losses by applying them separately to see how they influence the final image when used at different layers of the network. After that, we combine these losses to perform the complete process of neural style transfer, taking an input image and transforming it to reflect the style of another image while retaining its original content.

Part 1: Content Reconstruction

Description:

The content loss calculates the difference between the features of the input image and the target content image at a specific layer, using the squared L2 distance. This feature extraction is performed using a pre-trained VGG-19 network from torchvision.models, with its weights kept constant. To adjust the input image, we use the LBFGS optimizer, a type of quasi-Newton method.

In the experiments below, the content loss is applied after various layers, and the optimization process is carried out for 1000 steps.

Experiments:

Reflecting on the outcomes presented, it appears that incorporating content loss subsequent to the "conv_2" and "conv_4" layers yields the most effective results. Moving forward, the further results will introduce content loss following the "conv_2" layer.

Reconstruction

The results of the reconstructions look almost exactly like the content image, with minimal noise.

Part 2: Texture Synthesis

Description:

The style distance between two images can be assessed using the gram matrix, which captures the correlation across all dimensions by multiplying the feature map matrix with its transpose for each layer. The style loss is determined by calculating the squared L2 distance between the gram matrices of the input and target style images at chosen layers.

In the experiments that follow, I focused on optimizing the style loss at various layers, conducting the optimization process over 1000 steps.

Experiments:

I use the configuration where style loss is added after each of "conv_1, conv_2, conv_3, conv_4, conv_5"

Reconstruction

Beginning with varied noise values leads to distinct outcomes in the optimized styles. Although the generated styles share a thematic resemblance, they differ in their pixel-level details.

Part 3: Neural Style Transfer

Next, we merge the content loss and style loss to execute the neural style transfer process. Adjustments are made to the hyper-parameters, and the gram matrix is normalized across the feature pixels by dividing by the total element count in the feature map. The following experiment explores various weight settings for the style loss.

Experiments:

1. Style Weight

Following are some results with content_weight = 1 and varying style_weight

Other hyper-parameters chosen: num_steps=700; content_layers=[conv1, conv2, conv3, conv4, conv5]

2. Results Grid (2x2)

3. Random Noise

From a qualitative standpoint, the results obtained through content initialization appear superior to those achieved with random initialization. With content initialization, the structure of the content image is more effectively maintained in the output, and the applied style does not overwhelm the content. Additionally, in terms of convergence, content initialization reaches completion more swiftly than random initialization, as indicated by the observed runtimes.

4. Custom Style Transfer Images

---