Bells and Whistles attempted: Stylize grumpy cats + style transfer a video

Part 1

Effect of optimizing content at different layers

Content loss on different layers in order: conv1, conv2, conv3, conv4, conv5, conv8, conv12, conv16

We notice that content reconstruction with earlier layers work pretty well, and gradually with later layers more and more noise can be observed -- this might be because the earlier layers can learn more of the structural knowledge of the image.

Reconstructed Images compared with GT

We show the results with conv_3

We show the ground truth image on left and reconstructed image on the right. The results look very similar to the original image with a little bit of noise and some loss of details - for eg, the teeth of wally. The results with conv_1 are probably going to be even better but we discover in later exps that conv_3 works best when combined with style loss and hence we use conv_3 here

Part 2

Effect of Texture synthesis loss on various layers

Texture loss on different layers in order: conv1, conv2, conv3, conv4, conv5, conv8, conv12, conv16

We notice that texture reconstruction with earlier layers retain the texture but does not retain the structure very well, the middle layers do mantain both the structure and texture, while the later layers are almost pure noise

Reconstruted Images compared with GT

We show the results with conv_3

3.1 Style Transfer implementation details

We decided to use content_layer to be conv_3 and style layers to be an equally weighted combination of conv_1, conv_2, conv_3, conv_4. We do proper normalization of gram matrix by normalizing it with the number of pixels in the feature map. We find style_weight of 20000 to work best and content_weight of 1. We train for 300 iterations using the LBFGS optimizer

3.2 Mixing two content images with two style images

3.3 Comparision with content vs noise init

Below are the corresponding images with noise init

We notice that using content images works better -- it better preserves the details of the original content image. We also notice that the initialization with content image converges faster.

3.3 Stylize own images

Bells and Whistles

Stylize grumpy cats:

Stylize a video

We stylize the first 2 seconds of the video with a style -- we train for 50 iterations for each frame and we use the first 50 frames of the video due to computational requirements.