Learning Based Image Synthesis (16-726)

Assignment 4

Aviral Agrawal (avirala)

Bells & Whistles attempted

1. Overview

This assignment revolves around style transfer from a source 'style' image to a 'content' image. This 'artist' AI that we create is then tested on multiple images and across multiple styles. The work is also extended to video, and we give some artistic touches to some Grumpy cats!

2. Content Reconstruction

In this section we only implement "content loss" and reconstruct the "content" image from random input noise. The content loss is calculated as the mean squared error in feature domain between the source "content" image and the feature extracted from an input image. The input image starts from random noise but is iteratively updated to minimize the content loss. The feature extraction is done using a pre-trained VGG-19 network.


The following table shows the objective and qualitative results for the obtained reconstruction using varying conv-depths for feature extraction, subsequently used to compute the content loss:

conv-Depth	Reconstructed Image	Peak-signal-to-noise-ratio (dB)
conv-1		60.72
conv-4		14.68
conv-7		10.59
conv-10		8.65
conv-13		8.03
conv-16		7.70

From the above results we infer that the reconstructed image at very high conv-depths contains too much noise and not able to retain the desired information (content) from the "content" image. On the other hand, the reconstructed image at very low conv-depths is reconstructing the content too strictly. However, our end goal is style-transfer, hence, our reconstruction should be a bit flexible. Hence, we choose conv-depth = 3.
Thus, the best reconstructed image, obtained at conv-depth = 3 is as follows :


Now we investigate whether the image reconstruction is sensitive to the random noise provided as the input image.

Random Noise	Reconstructed Image	Peak-signal-to-noise-ratio (dB)
		20.27
		20.23

From the above results we infer that the image reconstruction is not sensitive to the random noise provided as the input image.

3. Texture Synthesis

In this section, we implement "style loss" and synthesize a "style" image from random input noise. The style loss is calculated as the mean squared error in feature domain between the source "style" image and the feature extracted from an input image. The input image starts from random noise but is iteratively updated to minimize the style loss. The feature extraction is done using a pre-trained VGG-19 network.


The following table shows the qualitative results for the obtained stylized image using varying conv-depths for feature extraction, subsequently used to compute the style loss:

conv-Depth	Stylized Image
conv-1, 2, 3
conv-1, 2, 3, 4, 5
conv-7, 8, 9
conv-14, 15, 16, 17, 18
conv-16, 17, 18
conv-1, 8, 16

From the above results we infer that the stylized image at very high conv-depths contains too much noise and not able to retain the desired information (style) from the "style" image. On the other hand, the stylized image at very low conv-depths is synthesizing the style too strictly, i.e., trying to reconstuct the actual features of the style image rather than the inherent style. However, our end goal is style-transfer, hence, our stylized image should be a bit flexible. Hence, we choose shallow conv-depths and also compute the style over a larger window of features = 1, 2, 3, 4, 5.
Thus, the best stylized image image, aggregated over a window of conv-depths = 1, 2, 3, 4, 5 is as follows :


Now we investigate whether the stylization is sensitive to the random noise provided as the input image.

Random Noise	Stylized Image

From the above results we infer that the stylized pattern remains the same given changes in the random noise provided as the input image. However, the location of the pattern features might be shifted spatially.

4. Style Transfer

In this section, we use the previously implemented "content loss" and "style loss" to perform style transfer from a source "style" image to a target "content" image. The total loss is a weighted sum of the content loss and the style loss. The input image starts from the target "content" image but is iteratively updated to minimize the total loss. The feature extraction is done using a pre-trained VGG-19 network.

Implementation Details The Gram matrix was normalized by the number elements in each feature map. So a feature map of shape a x b x c x d, where a is the batch dimension and b is the number of feature maps, the Gram matrix was divided by N = c x d. The content loss was implemented using a L2 loss, and the style loss was implemented as a L2 loss on the normalized Gram matrices. The optimization was run for a maximum of 300 iterations. Hyper-parameter Tuning In order to generate a desired mix of style and content, we perform Hyper-parameter tuning on the "style_weight" and "content_weight". These hyper-parameters have a direct impact on loss and hence, the finally generated image. The following figures show the qualitative results for the obtained stylized image using varying style (content weights set to 1.0):
We use the follwing content and style images for the hyper-parameter tuning:

Thus, from the above hyper-parameter tuning we choose the best style_weight = 1.0 and content_weight = 1.0. The final stylized image is as follows:
Some results on a 3x3 grid



Initialization from Noise vs Content
From the above observations we see that initialization with content image generates more sharper content that when initialized with random noise input. Thus, the images look subjectively better. In terms of running time, both the methods are comparable. Results on my own images

5. Bells & Whistles : Stylize some Grumpy Cats!

6. Bells & Whistles : Apply style transfer to a video