16726 Assignment 4

by Zi Wang (ziwang2)

Overview

In this project, I implement neural style transfer that keeps particular content information in a distinct artistic style. The method's input are a content image, a style image, and an additional input image. This input image is optimized to align with the content and style of the two target images within their respective distance spaces.

For Bells & Whistles, I finished 1. Stylize your grump cats or Poisson blended images from the previous homework. (2pts) and 2.Use a feedforward network to output style transfer results directly (8 pts).

Part 1: Content Reconstruction

In this part, I implement a content-space loss and optimize a random noise image based solely on content loss. The content loss is calculated as the L2-distance between a specific layer's feature of the input image and the corresponding layer's feature of the target content image. To extract image features, we employ a pretrained VGG-19 network. The table below outlines the specific layers where the content loss is applied.

Target:

Layers Used in ContentLoss	Noise 1	Noise 2
Conv_2
Conv_4
Conv_6
Conv_8
Conv_10
Conv_12

Based on the images generated from the experiment, it is evident that optimizing content loss at shallow layers tends to preserve fine details of the content image, including textures and edges. The generated image will closely mimic the detailed aspects of the content image. ptimizing content loss at deep layers focuses on preserving the high-level content and structure of the image.

Considering the trade-off between textures and high-level details, my favorite would be conv_4.

For the comparison, I think there is no big diference between the generated images from two random noises.

Part2: Texture Synthesis

In this part, I implement style-space loss, employing the Gram matrix to quantify the stylistic difference between two images. The Gram matrix serves as a representation of the correlations among the feature maps within a CNN. Following the approach of the preceding section, I select specific layers for the application of style loss.

Target:

Layers Used in ContentLoss	Noise 1	Noise 2
Conv_1, Conv_2, Conv_3, Conv_4, Conv_5
Conv_6, Conv_7, Conv_8, Conv_9, Conv_10
Conv_1, Conv_3, Conv_5, Conv_7, Conv_9

The ablation study reveals that the optimal position for incorporating style loss is subsequent to each of the initial five convolutional layers. Applying style loss after layers 6 through 10 fails to accurately produce the desired texture, as the convolutional blocks later in the CNN tend to capture more abstract and generalized patterns (Looks like there are some noises in the images). The texture we aim to replicate is highly detailed and specific, characterized by edges, corners, etc., necessitating its capture through the earlier convolutional blocks. Intuitively, the last row in the ablation study essentially reflects an average of the effectiveness with which the early and later blocks reproduce the target texture. Thus, the preferable choice is the first setup. Still, I didn't see any obvious difference between the generated images with different noise initializations.

Part 3: Style Transfer

In the Gram matrix, I normalized the matrix based on the number of pixels and ultimately selected the best-performing parameters as style_weight=1e5 and content_weight=1.

Here I combine different content images with different style images. Let's see the results!

Style\Content	dancing	fallingwater	phipps	tubingen
escher_sphere
frida_kahlo
starry_night
the_scream

Here I visualize the different results when the input_img is noise or content image.

	Input: Noise	Input: Content Image
escher_sphere
frida_kahlo
starry_night
the_scream

With a content image as input, the result of style transfer will exhibit more of the style, which aligns with our expectations. This is because, when the input is already a content image, the major component of the loss is the StyleLoss, thus it will focus more on the style aspects. Conversely, if noise is used as the input, then the loss will pay attention to both content and style, retaining more of the content part. The training times of these two kinds of input are quite similar and close to each other.

Here I try style transfer on some of my favorite images.

Content	Style	Result

Bells & Whistles (Extra Points)

Stylize your grump cats or Poisson blended images from the previous homework. (2pts)

Style\Content	content_1	content_2
starry_night
the scream

Use a feedforward network to output style transfer results directly (8 pts).

I modified DCGenerator in the previous assignment a little to do style transfer. Specifically, I modified the architechture parameters to make the network output an image with shape 512*512. Then I use Adam optimizer with learning_rate=1e-3 to optimize the Generator. I trained the network for 1000 steps. Please refer to my code run_GAN.py for more details. Here are some visualization results. We see that using a feedforward network is also capable of outputing style transfer results. (Reference: https://cs.stanford.edu/people/jcjohns/papers/eccv16/JohnsonECCV16.pdf)

Content	Style	Result