Assignment 4 - Neural Style Transfer

(Bells and Whistles at the end)

Part 1 : Content Reconstruction

Optimizing Content Loss at Different Layers

Here we attempt to reconstruct the original image by defining an L2 loss between the source (noise) and target (input) image. However, instead of directly minimizing the loss over the images, we pass the images through a pre-trained network and attempt to minimize the loss at different stages of the network.

Procedure:

1. Define source and target images
2. Download pre-trained network which acts as feature extractor
3. Get network output at L’th layer
4. Define L2 loss between L’th layer output of source and target image

L2 Loss $|f_X^L-f_c^L{|}_2^2$

Experiment : Optimizing Content Loss at Different Layers

The L2 minimization was done between model output of noise and content image at different convolution layers as shown below:

As seen from above, reconstruction works best at Conv2 and Conv4. In deeper layers, the some high frequency details are lost and the image appears grainy.

Experiment : Reconstruction from Random Noise

Taking two random images generated from a normal distribution as input, we reconstructed the original image using the L2 loss at Conv2.

As seen above, in both cases, optimizing at Conv2 allows us to retreive almost the original image.

Part 2 : Texture Synthesis

Texture synthesis relies on style loss, which quantifies the dissimilarity between the styles of two images. In the context of this assignment, the Gram matrix serves as a measure of style. The Gram matrix captures the correlation between channel dimensions for each pixel location (H, W).

Specifically, for the $L^{th}$ layer with feature image $f_k^L$, the Gram matrix is computed as:

$$G=f_k^L.(f_k^L{)}^T$$

Resulting in a shape of (N, K, K). The goal is to minimize the difference between the Gram matrices of our optimized and predicted features.

Experiment : Texture loss at different layers

The Style loss was then applied applied by combining the outpts at different layers. The results are shown below:

• From above images, it can be seen that Conv1 + Conv2 + Conv3 + Conv4 + Conv5 yields the best results.
• When combining earlier and deeper layers such as Conv1 + Conv2 + Conv5 + Conv8 + Conv9 we get similar style, but some colors appear incorrect

Experiment : Reconstruction from Noise based on Style Loss

• Starting from two images generated as random noise, we try to minimize the style loss
• I used the Conv1 + Conv2 + Conv3 + Conv4 + Conv5 configuration for style loss

The sythesized styles (‘noise image after optimization’) are globally similar but lack any pixel-to-pixel correspondence.

Part 3 : Style Transfer

Style Transfer is implemented by adding the Reconstruction and Style losses while optimizing the input image.

Experiment : Hyperparameter Tuning

For this experiment, I tried different weights on style loss while keeping content loss fixed. The results are shown below:

From the above examples, I think:

• Content weight = $1$
• Style weight = ${10}^6$

Would work best for style transfer

Experiment : Style Transfer Results on Different Images

Experiment : Style Transfer with Different Initializations

Initializing the input image was tested in two ways:

• Input image was same as content image
• Input image was random noise

Content initialization is clearly better in maintaining structure. Further the runtime was also considerably better for content intialization.

Experiment : Style Transfer on Favorite Images

Bells and Whistles

Applying Style Transfer on Outputs of Previous Homeworks

I used the images from the poisson blending task below:

---