In this assignment, the main topic is neural style transfer, a technique that allows you to generate images that combine the content of one image with the style of another. The assignment is divided into three main parts:
1. Content Space Optimization: Starting from random noise, optimize the image to match the content of a given content image. This process involves minimizing the content distance between the generated image and the content image. Its focus is in how to optimize pixels to match a target content.
2. Texture Generation: In this part, it wll focus solely on generating textures. Ignoring the content, optimize the image to match the style of a given style image. This involves understanding how to capture style information using techniques like the Gram matrix, which represents the correlations between different feature maps in a convolutional neural network.
3. Neural Style Transfer: Finally combining the concepts from the previous two parts to perform full neural style transfer. Here, I will optimize an input image to simultaneously match the content of one image and the style of another. This involves defining loss functions that balance content and style distances and then using optimization algorithms like gradient descent to iteratively update the input image until it closely resembles the desired output.

The feature set of a given input image, represented as X, at the L-th layer is noted as F^L_X, while for the chosen target content image it is noted as F^L_C. The task's content loss is computed as the aggregate of the squared L2-norm distance among the chosen features of these two images:

L_content = Σ_{l ∈ Γ} || F^l_X - F^l_C ||₂²,

where \( Γ \) signifies the collective layers within the model designated for content feature extraction. Specifically, for this task, each initial layer within the convolution blocks of VGG19, named \( \texttt{conv1_1} \) through \( \texttt{conv5_1} \), is employed for extracting features.

Content reconstruction has been executed across five different content images, each processed with two distinct noise inputs. The evaluation of results utilized diverse convolution blocks from VGG19, with the loss metrics.

1. Report the effect of optimizing content loss at different layers. [15 points]

The graph displays the optimization of content loss at various layers of a convolutional neural network over several iterations. Initially, the first layer shows the highest loss but decreases rapidly, suggesting efficient learning at this level. Higher layers begin with lower initial losses but don't improve as quickly, indicating they may learn differently or require alternate optimization strategies. The earlier layers, especially the first, appear most effective at reducing content loss, which could imply they are more crucial for content reconstruction in the initial optimization phases.

It's apparent that a less complex network structure, indicated by lower content layers, enhances reconstruction effectiveness, both regarding the speed of convergence and the fidelity of the resulting image. This could be due to the increased instability presented by more complex networks, potentially leading to significant data loss and entrapment in local minima, which in turn may impede the convergence process.

2. Choose your favorite one (specify it on the website). Take two random noises as two input images, optimize them only with content loss. Please include your results on the website and compare each other with the content image(15 points)

I selected the fourth convolutional layer (conv4) as the content representation layer due to its superior performance in achieving desirable results within a limited number of iterations, compared to other layers. Although the differences between the outcomes produced by various layers are subtle at first glance, a meticulous examination through the subtraction of resultant images from each layer revealed distinct variations. This methodical approach allowed for a nuanced assessment of the layers' efficacy in content representation, ultimately guiding the decision to utilize conv4 for its optimal balance of detail retention and computational efficiency.

To determine the style loss, which is a measure of how much the style of our input image differs from that of our target style image, we first rearrange each feature into a two-dimensional format by combining the batch size and channels, and then the height and width. Once we have this new arrangement, we compute what's known as the Gram matrix. This matrix is a special representation that captures the essence of the style of the image by considering how features across the channels relate to each other.

The next step in calculating the style loss involves normalizing this Gram matrix based on the number of elements in each feature, which takes into account the batch size, the number of channels, and the dimensions of the image. After normalization, we compare the Gram matrix of our input image with that of our style target. The style loss is then computed by taking the sum of the differences between the corresponding elements of these two Gram matrices across all layers specified for style extraction. This sum gives us an overall value that represents the disparity in style between the two images.

Report the effect of optimizing texture loss at different layers. Use one of the configurations; specify it in the website and: [15 points]

The graph shows optimization trajectory of style loss for five different convolutional layers in a neural network as the training progresses through iterations ranging from 50 to 300. The vertical axis measures the style loss, likely on a logarithmic scale, which allows us to appreciate changes across a wide range of values. The horizontal axis enumerates the training iterations. Here's a more detailed interpretation: Conv_1 layer starts with the highest style loss, but all layers show a steep decline initially, which indicates a rapid learning phase where the network quickly adjusts to capture the style features. As the iterations progress, each layer shows diminishing style loss, with the rate of reduction slowing down after the initial drop. The final style loss values across the layers are very close, suggesting that each layer, regardless of its position in the network, has been similarly tuned to the style aspects of the training data by the end of the optimization process. The overall takeaway from the graph is that the network learns most about the style in the early stages of training and that subsequent gains from additional iterations are marginal, pointing to an efficient optimization process that rapidly reduces style loss across all layers.

Upon examining the images derived from the 50th iteration, it is evident that style loss computed at earlier layers yields superior results at an accelerated pace. This observation aligns with the expectation that initial layers in a network capture more granular and fundamental stylistic elements, thereby facilitating more rapid convergence towards the desired aesthetic outcome.

Take two random noises as two input images, optimize them only with style loss. Please include your results on the website and compare these two synthesized textures. [15 points]

Although they are mostly same, in the upper portion of the image, starting from the leftmost panel, there is a near-random distribution of color with minimal discernible structure, indicative of an early stage in the learning process. Moving towards the right, we observe the emergence of more coherent patterns and a gradual development of structure, likely representing successive iterations. Conversely although difference are minute, The lower portion of the image begins with a different noise pattern, which may have different initial distributions or spectral characteristics. As we progress horizontally, the style seems to evolve little different compared to the upper section. It suggests that the starting noise can influence the trajectory of the learning process, even when converging towards a similar style.