**Learning-Based Image Synthesis** **Assignment 4** - Neural Style Transfer Overview ============================================================== The goal of this assignment is to perform Neural Style transfer! Style Transfer means creating a new image by taking content from one image and style from another image. The algorithm takes in a content image, a style image, and another input image. The input image is optimized to match the previous two target images in content and style distance space. We will sequentially do content reconstruction first, next texture synthesis and finally piece all the elements of style transfer together. Some cool results are provided at the end of the webpage. Content Reconstruction ============================================================== Here we implement content-space loss and optimize a random noise with respect to the content loss only. We use MSE loss as the content loss between the features from the input and the target image. We use few initial layers from the pretrained VGG-19 network as feature extractors and apply the content loss on top of it. For the optimization, we use a quasi-newton optimizer, LBFGS optimizer. Effect of optimizing content loss at different layers -------------------------------------------------------------- Shown below is the target image and below that are the output images reconstructed using content loss from features at different layers of the VGG network.    | Layer | SSIM | |----------|---------| | conv_1 | 0.9021 | | conv_2 | 0.9013 | | conv_3 | 0.7362 | | conv_4 | 0.7049 | | conv_5 | 0.4332 | | conv_7 | 0.2607 | As can be seen from the reconstructed images, it becomes more noisy when more deeper conv layers are used. This is validated in the SSIM metrics shown above too. The SSIM is the image similarity metric between the reconstructed images and the target image. The SSIM metric is the best in the initial layers but taking conv_1 might give only high level features since the initial layers focus too much on the high level global structure and features. I take conv_3 for the further experiments. Optimizing random noises with content loss --------------------------------------------------------------   | Images | SSIM | |----------|---------| | Reconstructed image 1 & targeted image | 0.8126 | | Reconstructed image 2 & targeted image | 0.8387 | | Reconstructed image 1 & Reconstructed image 2 | 0.8554 | Shown above are the image reconstructions from two random noises. It looks like it has reconstructed well and the random noise doesn't matter because the reconstructions from it look similar. This is also validated from the SSIM values. Texture Synthesis ============================================================== Next we implement the style-space loss using Gram matrix. Gram matrix is the correlation of two vectors on every dimension and the gram matrices of the input image and the target image has to be as close as possible. We reuse the VGG-19 network as the feature extractor and add the style loss after it. The optimization is done only on the style loss. Optimizing texture loss at different layers --------------------------------------------------------------    Shown above are the reconstructed images using style loss with features extracted from different combinations of the conv layers from the VGG network. In general, using the initial layers of the network seems better visually than the deeper layers which tend to be more noisy. conv_1 - conv_3 and conv_1 - conv_5 seem to be learning the style well and between these two, I use conv_1 - conv_5 in the further sections because I feel it is only capturing the style and not some parts of the content such as the black regions seen in the conv_1 - conv_3 image. Optimizing random noises with content loss --------------------------------------------------------------   The two reconstructed images from the random noises seem very similar to me and the style looks very close to the style in the target image indicating that the style optimizations is happening properly and it doesn't matter what the noise is. Style Transfer ============================================================== We piece all the segments together and perform style transfer here. The feature extractor from the VGG network will have both the content and style loss layers and the otpimization will be on both these losses using a weighted procedure. Hyperparameter tuning and implementation details -------------------------------------------------------------- As discussed in the previous sections, I chose conv_3 layer to optimize the content loss and conv_1 - conv_5 to optimize the style loss. I normalize the values of the gram matrix by dividing by the number of element in each feature maps being fed to it. The features being fed into the gram matrix will have shape a x b x c x d. I first reshape it into (a*b) x (c*d) and divide the gram matrix obtained by a*b*c*d. The number of optimization steps used is 300. The weights assigned to content and style loss are based on the value range that they have. I keep the content loss weight at 1 and vary the style loss weight. Shown below are the results using different style loss weight.   I think style weight = 1000000 gave the best image with style transferred but content preserved. Hence I go ahead with that! Results -------------------------------------------------------------- | | | | |----------|---------|---------| | | | | | | | | Noise vs content image --------------------------------------------------------------   | Image | time taken (s) | |----------|---------| | random noise 1 | 5.94 | | content image - dancing | 5.92 | | random noise 2 | 5.85 | | content image - wally | 5.83 | Shown above are the results when random noise is taken as input image vs the content image for dancing and wally content images with starry night style image. Taking random noise, though the content is seen in the resulting image, is not very evident and is noisy. Hence, taking the content image as the input makes sense visually. Time wise, as shown both the methods take almost the same time. The time mentioned is when the algorithm is run on an NVIDIA A100 GPU for 300 iterations. Results on my favorite images -------------------------------------------------------------- | | | | |----------|---------|---------| | | | | | | | | I have applied starry night and the scream styles on an elephant from Kerala in India and Lake Erie which borders many states in North America. Bells & Whistles ============================================================== Stylize your grump cats -------------------------------------------------------------- | | | | |----------|---------|---------| | | | | | | | |