Welcome to the webpage for Kevin You's submission for project 4.
In this assignment we perform neural style transfer. We begin with a pretrained VGG19 network. Given a content image and style image, the output image is optimized with the reconstruction loss against the content image (L2 distance of feature maps) and the style loss against the style image (L2 distance of gram matrix of feature maps).
We first implement content reconstruction. I have used a hyperparameter of lr = 0.01, and the initial image is randomized to the range [0,1). For content reconstruction 80 steps is more than enough. Results are shown below.
| Real | Conv2 | Conv4 | Conv6 |
|---|---|---|---|
|
|
|
|
It is unsuprising that the furthur the layer is used, the blurrier the image becomes. It is also worth noting that the image's color isn't correct. This is due to the normalization layer at the beginning. While normalization doesn't help us with content reconstruction, it can help us layer for style transfer, intuitively since it gives us more flexibility.
| Conv2 | Conv4 |
|---|---|
 |
 |
I opted to stick with the default conv4 layer since it seemed to give decent reconstruction, while not being as restrictive as conv2. Indeed, see this example for style transfer, where the style choices are fixed, but the content layer uses conv2 and conv4. I like the bolder edges on conv4, that draws colors from the starry night.
| Noise 1 | Noise 2 |
|---|---|
 |
 |
 |
 |
 |
 |
Unsuprisingly, these two images look almost exactly the same. (Though they are different, and one can check by looking at the top-left most pixel). Shown in last row is difference to original image.
Next we do texture synthesis. The hyperparameters are the same, except that I decided to be a bit more generous and increase runs to 100. I considered conv layers 1,2,3,4,5,6,8 (rather arbitrairly), and picked some consecutive sequ
| Real | Conv1,2,3 | Conv1,2,3,4,5 | Conv3,4,5 | Conv5,6,8 |
|---|---|---|---|---|
 |
 |
 |
 |
 |
We see that the first image reproduces the colors, but lacks the long strokes from the original image. Indeed, we need to go deeper in the model to obtain pixels with larger receptive fields. However, if we go too deep, the color starts becoming distorted. Therefore, I ended up sticking with the default of conv1,2,3,4,5.
| Noise 1 | Noise 2 |
|---|---|
 |
 |
 |
 |
Nothing much to see from the two noise images. It is good to know that we can generate random textures though.
Now we finally experiment with the hyperparameter style_weight (assuming content_weight = 1). We keep previous hyperparameters the same.
| 5000 | 50000 | 50000 content init | 200000 |
|---|---|---|---|
 |
 |
 |
 |
Not much to say here. 50000 worked best for me. The others had too little style or content. It is worth noting that this number will depend on the layers being used. Also I have used one instance of initializing the input image by the content image. The speeds were roughly the same, around 2 minutes, though the one with the content initialization had a color spectrum closer to the original image (keeping in mind of normalization).
|
 |
|
|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |