1. Overview
1.1 Introduction
Neural Style Transfer (NST) is a fascinating technique in the field of computer vision and deep learning, allowing the synthesis of a new image by applying the style of a reference image to the content of a target image. This process leverages the features extracted by deep neural networks to blend two images artistically.
The key components in NST include the content loss and style loss computations. Content loss measures how much the content of two images differ, typically using the squared error between feature representations of the content and generated images. Mathematically, it's \(L_{content} = \frac{1}{2} \sum (F_{ij}^l - P_{ij}^l)^2\), where \(F^l\) and \(P^l\) are the feature representations of the generated and original content images at layer \(l\). Style loss is computed as the mean squared error between the Gram matrices (G) of the style and generated images' feature representations, expressed as \(L_{style}^l = \frac{1}{4N_l^2M_l^2} \sum (G_{ij}^l - A_{ij}^l)^2\), where \(N_l\) and \(M_l\) denote the number of filters and the size of the feature map, respectively, for layer \(l\). The total loss is a weighted sum of content loss and style loss, optimized to generate the final stylized image.
The computation of content and style losses can be performed at different layers of a convolutional neural network, each offering unique perspectives on the image's content and style. For content reconstruction, deeper layers are preferred as they capture higher-level content details, whereas for style synthesis, losses are computed across multiple layers to capture a comprehensive style representation. This layer-wise approach allows for a more nuanced control over the balance between content preservation and style incorporation, enabling the generation of images that are both recognizably similar to the content image and stylistically vibrant.
This assignment comprises four main parts: content reconstruction, where the focus is on replicating the content of the target image; style synthesis, which emphasizes on capturing the essence of the style image; style transfer, combining the previous two tasks to create an image that mirrors the content of one image and the style of another; and lastly, bells and whistles, which expands on the basic style transfer task. The bells and whistles include performing style transfer on a grumpy cat image, integrating Poisson blending to improve the coherence of the transferred style, and applying style transfer to video sequences for dynamic stylization.
2. Implementation
Part 1: Content Reconstruction
Optimizing content loss at different convolutional layers of a neural network produces varied effects on the synthesized image. As observed in the provided image series, optimization at initial layers like Conv1 and Conv2 preserves fine details and textures, closely resembling the original content. With mid-level layers such as Conv3 to Conv5, there is a subtle abstraction, yet maintaining recognizable structures and spatial hierarchy. Higher layers, Conv6 and beyond, lead to a progressive loss of detail and an increase in high-level features and abstraction, causing a significant distortion in the image, which becomes more pronounced in Conv9 and Conv10. This progression illustrates how earlier layers capture pixel-level accuracy while deeper layers represent the semantic content.
Among the ten layers showcased, Conv4 stands out as an optimal choice, as it maintains content integrity while imparting a subtle stylistic transformation, indicating a harmony between detail preservation and feature abstraction that is aesthetically pleasing and true to the original image's content.
In the experiment where two different random noise images (Noise1 and Noise2) are used as inputs and optimized solely with content loss, the results look exactly the same, both closely resembling the content image. Despite the initial dissimilarity of the noise inputs, the optimization process guided by content loss converges to a similar solution for both, focusing on reducing the pixel-level differences between the generated image and the content image. The consistency in results underscores the robustness of the optimization algorithm, which effectively ignores the initial noise patterns and converges to a representation that captures the essence of the content, demonstrating the power of content loss in driving the transformation of even a highly random input towards a coherent content structure.
Part 2: Texture Synthesis
The texture synthesis results from the different layer configurations exhibit distinct characteristics. The Conv12345 configuration captures a wide array of textural information, translating into a nuanced and visually coherent style that retains a significant amount of the original image's color dynamics and brushstroke patterns.
Conversely, Conv678910, which utilizes deeper layers, seems to encode higher-level features that contribute to a greater degree of abstraction and a departure from the original style's intricate details. The resulting texture is less defined and more chaotic, with the style's essential qualities being somewhat obscured.
Between Conv13579 and Conv246810, there is a subtle difference in the texture synthesis. Conv13579 tends to preserve more of the fine details and the vividness of the style image, likely due to the inclusion of lower layers that capture basic textures. In contrast, Conv246810, while still presenting a complex texture, appears slightly more abstract, potentially because it skips every other layer, missing some of the detailed texture information captured by the omitted layers.
When comparing Conv12345 and Conv13579, there is a choice between a more coarse or a more nuanced texture synthesis. Conv12345 offers a balance that can adapt well to both the content and the style, providing a synthesis that respects the granularity of the original style. Conv13579, skipping the even layers, might result in a texture that lacks some of the continuity and smooth transitions present in Conv12345.
After evaluating all configurations, Conv12345 is selected as the most ideal for texture synthesis. This configuration successfully synthesizes a texture that is rich, detailed, and most true to the original style, making it the preferred choice for demonstrating the capabilities of neural style transfer in capturing and reapplying texture effectively.
When two distinct random noise images (Noise1 and Noise2) are optimized using only style loss, the outcome textures are very similar. Both synthesized textures display the same overarching patterns, color schemes, and visual rhythms, suggesting that style loss optimization powerfully guides disparate inputs to a unified style representation. This process exemplifies the capacity of neural style transfer to abstract the essence of a style and apply it consistently, regardless of the initial state of the input image. The two textures, while starting from unique noise patterns, converge to similar artistic interpretations under the directive force of the style loss function.
Part 3: Style Transfer
Gram matrix normalization is conducted within the gram_matrix function, ensuring that the style features are scaled correctly to contribute evenly across all layers. This normalization is vital for preventing the initial layers from having an oversized impact due to their larger feature maps.
The hyper-parameters were meticuslously tuned to achieve an optimal balance between content and style. The number of steps for the optimization process was set to 400, providing sufficient iterations for convergence yet avoiding too much that make loss shoot to nan. The weight ratio between style and content loss was carefully adjusted, starting from a baseline of 1,000,000:1. Through experimentation, these weights were fine-tuned to harmonize the contributions of style and content to the final image for most cases.
The code logic is organized with a main function serving as the entry point, which calls the run_optimization function to perform the style transfer. The style and content losses are encapsulated within their respective classes, ContentLoss and StyleLoss, each implementing a forward method. This design allows the loss modules to behave like transparent layers within the network, computing the loss while passing through the input.
To integrate the loss modules within the VGG19 network, the get_style_model_and_losses function inserts these modules in place, directly after the corresponding convolutional layers. This insertion relies on matching the layer names, ensuring that each content and style loss module is aligned with its intended convolutional layer within the architecture. The Sequential model is built dynamically, layer by layer, with the content and style loss layers added immediately following their targeted convolutional layers.
Grid results optimized from 3 content images mixing with 3 style images:
Comparing the results of style transfer using random noise and a content image as initial inputs reveals significant differences.
The running times for both processes were comparable, with content image initialization slightly faster, which might be attributed to the pre-existing structure in the content image allowing for quicker convergence of the optimization algorithm. As demonstrated by the console output, the total loss for content initialization drops more swiftly than with noise, suggesting a more efficient path towards minimizing the loss functions.
In terms of image quality, content image initialization retrains the content picture more as opposed to smashing everything on the picture with style image in the noise initialization. It leads a better-defined subject, finer details, and a more cohesive integration of style and content. Therefore, content image initialization not only enhances the quality of the synthesized image but also marginally improves the efficiency of the style transfer process.
Try style transfer on some of your favorite images:
Bells & Whistles
1. Stylize your grump cats or Poisson blended images from the previous homework. (2pts)
2. Apply style transfer to a video. You could try frame by frame method (2pts) or applying temporal smoothness for better output(4pts).
Frame by frame method
The frame by frame video method assembles a sequence of still images, displaying them in quick succession to produce the illusion of motion. The style transfer is done exactly the same as in the image case.
Applying temporal smoothness for better output
Incorporating temporal loss into the style transfer process enhances video consistency. As can be seen from the video, while overall jitter is not ostensibly reduced, it is clear that without applying temporal loss, there was a blackout in the video, but now there isn't. Temporal loss is calculated by the mean squared error between the current frame's input and the previous frame's output. There is potential to expand it into considering multiple previous frames.