This project aims to utilize pretrained model to perform neural style transfer which resembles specific content in a certain artistic style. In other words, the system is able to generate an image with given content and artistic style. The system takes an input image, a content image and a style image as input, and generates the corresponding image with certain content and artistic style. The main idea of the system is to optimize the input image's content and style feature space to match the style and content feature space of the images the user provided.
The system is based on a pretrained VGG-19 model (pretrained on the imagenet). Both content and style feature can be extracted from the outputs of the some block of the pretrained VGG-19. The following two paragraphs will elaborate on how to extract content and style features.
For optimizing the content features.
For optimizing the style features.
In this project, I used a quasi-newton optimizer LBFGS to optimize the image.
Here are some examples of the reconstructed (from random noise) images based on content features extracted from output of different blocks (using dancing.jpg and phipps.jpeg from the content folder). For each block, I will show two different images that are being reconstructed and discuss the effect of optimizing content loss at different layers. After that I will compare images reconstructed based on different random noise.
Original:
1st block:
2nd block:
3rd block:
4th block:
5th block:
Discussion:
From the results above, we could see that the first two layers keeps most of the information of the original content images, since it reconstruct the images pretty well (with little noise).
For the 3rd block, we could see that the content feature space kept most of the layout, color, edges, most of the texture information of the original image. But it lacks texture details at a granular level.
For the 4th block. the content feature space only keeps the overall layout of the original images (one can roughly recognize what the object is in the certain reconstructed image), it does not have any color information, neither texture information.
For the 5th block. I think that the model only keeps the information that can help it for image recognition. there exists some edges, but I think that it is hard for human beings to recognize what these edges are.
The following are the images reconstructed with two different random noise using features from 3rd layers, with random seed 715 and 725, respectively
I think these two reconstructed images are roughly the same as expected. Since I use the 3rd layer features (which contains most information from the original image, thus the system will try to recover the original image, which means that the initial random noise will not have a large impact on the result), the result should be same regardless of different initial random noise.
Here are some examples of the generated textures (from random noise) images based on style features extracted from output of different blocks (using the_scream.jpeg and starry_night.jpeg). For each block, I will show two textures generated based on the images just mentioned and discuss the effect of optimizing style loss at different blocks. After that I will compare two textures generated based on two different initial random noise.
Original:
1st block:
2nd block:
3rd block:
4th block:
5th block:
All blocks:
Discussion:
For the first block, I think the feature space mainly contains the overall color theme of the style image, since the generated textures are mainly color blocks.
For the second block, I think the feature space mainly captures the styles of upper level color blocks (like thick brush, thick lines together with their corresponding colors in the style image)
Feature extracted from this block contains less color information, but contains more local pattern (thinner line style and pattern at a more granular level).
For the 4th and 5th block. The feature extracted here may contains texture at a even more granular level. In addition, these feature contains very few or even no color information, just some more granular "line style".
The following are the textures generated from two different random noise using features from all blocks outputs, with random seed 715 and 725, respectively
Though using the same style features, the generated images are different in content while having same style as expected. Since the style loss only optimize the "style" of the image and do little about the content. The content of the image here is determined by the initial random noise. Thus if the initial random noises are different, then the generated images will have different content but with same style.
I choose different hyper-paramters for generating images from random noise/content image (Gram matrix are normalized for both)
For content image:
image size: 256 (smaller edge of the image)
num_steps: 100
random seed: 715
content block: 4 content weight: 1
style block: 1,2,3,4,5 style weight 1e6
For content random noise:
image size: 256 (smaller edge of the image)
num_steps: 100
random seed: 715
content block: 4 content weight: 100
style block: 1,2,3,4,5 style weight 1e6
The following shows the images generated from random noise and content image
Comparison
Obviously, results generated from content images has much better quality than those generated from the random noise. The results generated from the content images keeps the content better, while the ones not (more like a bunch of textures that has a layout of the content). In addition, the results generated from random noise has a lot more noise and distortion than those generated from the content images. Moreover, the results that generated from content images has a better appearance of the "style". I ran all the test on my laptop 3080ti via Windows Subsystem for Linux (WSL), the running time of generating results from random noise and content images are rougly the same.
Style transfer on favorite images:
