_________________________________________________________

HW4 - Neural Style Transfer

Introduction

The objective of this project is to implement a neural style transfer algorithm capable of generating a stylized image that combines the content of one image with the artistic style of another. This process entails optimizing either a randomly initialized image or a duplication of the content image to minimize differences in both content and style representations, as defined by specific layers within a neural network.

The project is structured into three main parts. Initially, we focus on constructing the content loss function, which involves adjusting a content representation—starting from random noise—to closely match a given content image. This phase emphasizes the effectiveness of various convolutional layers within the VGG19 network in reconstructing image content. Subsequently, the exploration shifts towards texture synthesis through the application of style loss, aiming to generate textures from random noise that emulate the style of a provided image. The final segment integrates both content and style adjustments to perform full neural style transfer, showcasing a range of outcomes and conducting an analysis on the impact of layer selection.

Part 1: Content Reconstruction

Content loss quantifies the difference in content between two images, assessed at a specific layer within a neural network. In this segment, the VGG19 model serves to extract distinctive features from the images. The process begins by generating a reconstruction image from random noise, each instance initialized with a unique seed, and then optimizing this image within the pixel domain. For selected layers, features from both the original content image and the reconstructed image are extracted. The mean squared error (MSE) between these feature sets serves as the content loss. This calculated loss is then used to perform backpropagation through the network, refining the reconstruction image. To facilitate the computation of content loss, a dedicated content_loss layer is introduced following each convolutional layer in the model.

Report the effect of optimizing content loss at different layers.

The first table presents the reconstruction outcomes for the image "fullingwater image" when utilizing various layers.

into

conv_1 conv_2 conv_4
conv_6 conv_8 conv_10

The following table shows another reconstruction samples

into

conv_1 conv_2 conv_4
conv_6 conv_8 conv_10

Content reconstruction seeks to refine a pattern of noise until it closely resembles a specific input content image. This technique adjusts the initial noise to align with the characteristics of the content image, leveraging a mean squared error (MSE) loss function. To ensure the integrity of the VGG-19 model throughout the optimization process, the 'detach' method is utilized, which prevents the model from undergoing any updates.

The success of content reconstruction is influenced by the layer depth at which comparisons are made. In the initial layers, specifically layers 1 and 2, the resemblance between the reconstructed and the original content images is striking, indicating a high degree of content accuracy. On the contrary, selecting deeper layers for comparison tends to introduce more noise into the reconstructed image. This observation led to the selection of the Conv_4 layer as the preferred choice for extracting content features. The rationale behind this selection is the Conv_4 layer’s efficiency in preserving vital content details while simultaneously reducing the impact of the original image’s texture and style.

Part 2: Texture Synthesis

This section delves into texture synthesis using the VGG19 model as the sole mechanism. The process involves running both the reference style image and the target generated image through VGG19. At select layers within the model, the feature discrepancies are evaluated via the Gram matrix method. The Gram matrix is selected for its proficiency in capturing the image's textural or pattern-based attributes rather than the specific content, differentiating it as a tool for stylistic analysis. This square matrix is produced by multiplying a feature matrix with its transpose, thereby capturing the statistical relationships present among the image's feature maps.

The methodology and implementation for this part of the project are reflective of those discussed in Part I, augmented by the incorporation of style_loss layers. These layers are assimilated similarly to the previously mentioned content_loss layers, aiding in the optimization process that targets style replication.

Report the effect of optimizing texture loss at different layers.

The first table shows the texture synthesis results of frida_kahlo image using different layer.

into

conv_1 conv_2 conv_4
conv_6 conv_8 conv_1&2&3&4&5

The following table shows the texture synthesis of starry_night.jpeg.

into

conv_1 conv_2 conv_4
conv_6 conv_8 conv_1&2&3&4&5

Observations reveal that textures synthesized from the shallower layers of the network often appear blurry, missing the clear shapes and strokes that define the original image. These initial layers tend to capture only the most basic details of the texture. In contrast, when texture synthesis is performed using the network's deeper layers, the resulting textures include more complex details but suffer from an increased amount of noise, compromising the texture's integrity. To overcome these challenges, adopting a strategy that combines multiple layers—specifically, conv_1, conv_2, conv_3, conv_4, and conv_5—is recommended. This approach ensures that the resulting texture not only maintains the core attributes of the original texture but also exhibits a satisfactory level of quality.

Part 3: Style Transfer

Tune the hyper-parameters

I choose to use conv_4 for the content loss and conv_1&2&3&4&5 for the style loss. Then, I do the following ablation study, to tune the parameters of the style weight and content weight.

content image style image

The following is the ablation study of the weight for content loss and style loss (fix the content weight = 1):

style loss weight = 100 style loss weight = 1000 style loss weight = 10000 style loss weight = 100000
style loss weight = 175000 style loss weight = 250000 style loss weight = 500000 style loss weight = 1000000

The ablation study conducted highlights the selection of style loss weight at 175,000 (occasionally adjusted to 100,000) and content loss weight at 1. Within this configuration, the outcomes appeared most balanced, effectively preserving the content's integrity while also closely mirroring the texture of the style image.

My 3x3 grid results - from random noise

content images style image 1 style image 2

My 3x3 grid results - from content inital

The table illustrates the outcomes when utilizing the content image as the starting point for generation, facilitating a comparison with results generated from an initial state of random noise. It should be noted that the captions for the images were mistakenly labeled as "output image from noise" and should correctly read "output image from content," indicating an error in the captioning process.

content images style image 1 style image 2

When performing style transfer using a noise image versus a content image, the outcomes vary significantly, as illustrated by the two tables above. Utilizing a noise image for style transfer typically yields images of lower quality, characterized by blurriness, pixelation, and the presence of artifacts. On the other hand, starting with a content image leads to higher-quality results, with more distinct and recognizable features, allowing for a clearer preservation of the original content.

Regarding computational efficiency, the process of style transfer from a noise image takes, on average, 25 seconds on my device, whereas starting from a content image reduces the time to approximately 21 seconds. This indicates a computational advantage, as using a content image as the starting point doesn't require optimizing every pixel from scratch, leading to a quicker generation process.

Style transfer on my favorite other images

content image style image generated from content image