Content Reconstruction
Minimizing $L_{content}$ in different layers
We can observe that as the content loss layer becomes deeper in the VGG network, the resulting image becomes more similar to noise and less similar to the original image. My favorite is the first convolutional layer in the second block, as it retains most of the texture in the original image while not being overly similar to the original content image. This balance between texture and style preservation seems desirable.
MaxPooling causes information loss
The images in the previous section all have the same loss value(~0.5). But if we optimize the image for a specific number of steps, an interesting phenomenon is that the reconstructed image becomes significantly different after the first MaxPooling layer in the VGG network. Additionally, more steps are required to reach the same loss. This makes sense, as the MaxPooling layer discards information by only preserving the largest values, making it challenging to reconstruct the details from the network features.
Only the head matters in the last layer
Another interesting observation is that although the last reconstructed image appears mostly like noise, we can still discern a blurry dog head from the noise. This suggests that the VGG model learns meaningful features that are understandable by humans, and simply passing a "head" image to the model can produce the same CNN feature at the last layer.
2 Random noise, similar result
Despite the different initial noise, the final reconstructed images are quite similar, suggesting a degree of robustness in the reconstruction process.
Texture Synthesis
Minimizing $L_{style}$ in different layers
The middle layers produce more color-accurate styles and can capture the main patterns in the style image, such as the spiral-like shape in the Starry Night. This is surprising, given that the gram matrix computation requires the model to learn a form of matrix decomposition to reconstruct the pixel patterns from the inner products of activations.
Combination of layers produce more complex patterns
Experimenting with different combinations of layers for style synthesis reveals that using multiple layers produces much more complex and less noisy styles. The patterns generated from the first 4 layers are particularly intriguing and beautiful.
2 Random noise, different result
The style images generated from different random noise inputs are quite different, even though they have the same gram matrix. This suggests that the "pattern" in the original image can be reconstructed in various ways with the same gram matrix, as discussed in [2017-Risser_et_al].
Different weights
When normalizing the gram matrix and using 4 layers, I found the optimal style weight factor to be around 1e-2 to 1e-1. Larger or smaller values lead to more noise-like patterns.
Style Transfer
Style Transfer
Evaluating style transfer is subjective, as the optimal result may depend on the specific content and style images. In general, I use a small style and content weight (around 0.2) and a small learning rate (0.1), which leads to more stable results. The gram matrix is also normalized. For images with many details, it is better to use the first convolutional layer for $L_{content}$ to avoid losing important details at deeper layers.
Different Input
Trying both noise and content image as input, I found the results to be very similar, with the runtime also being comparable (less than 1 second difference).
My favorite image
The universe!