This homework aims to implement neural style transfer which resembles specific content in a certain artistic style.
I chose dancing.jpg to show the results of the optimizing content loss at different layers.
| Seed: 0 | Seed: 1 | |
|---|---|---|
| conv_4 | |
|
| conv_6 | |
|
| conv_9 | |
|
From the previous results, we can see that optimizing content loss on the lower layers maintains content with higher fidelity, but this effectiveness diminishes more at the higher layers like conv_9. The result was tested on two random noises (seed 0 and seed 1). From this we can see that the influence of noise appears quite minimal.
I chose picasso.jpg to show the results of the optimizing texture loss at different layers.
| Seed: 0 | Seed: 1 | |
|---|---|---|
| conv_4 |  |
 |
| conv_6 |  |
 |
| conv_9 |  |
 |
From the previous results, we can see that optimizing texture loss on the higher layers show more details of the styles in a more dense way. Optimizing on middle layers like conv_6 appears a very similar style to the input image. The result was tested on two random noises (seed 0 and seed 1). From this we can see that the influence of noise appears quite minimal.
Time to put pieces together! I chose dancing.jpg and fallingwater.png as input content images, and chose picasso.jpg and frida_kahlo.jpeg as input texture images. The content loss module computes the difference in content between two images at a specific network layer. It measures the mean squared error between the feature maps of the input image and a pre-defined content image, weighted by a factor. The feature maps from the content image are provided during initialization. Style loss calculates the loss related to the image's style. It uses the Gram matrix, derived from the feature maps, to quantify style differences. The Gram matrix is normalized to ensure equal contribution across layers, crucial for capturing style accurately.
For hyperparameter, I chose style_weight to be the default value of 100000.
First for input as random noise, the 2x2 grid of results is shown below.
 |
 |
|
|---|---|---|
 |
 |
 |
 |
 |
 |
Then for input as content images, the 2x2 grid of results is shown below.
|
|
|
|---|---|---|
|
 |
 |
|
 |
 |
Personally, think the image quality is better when the input is random noise with stronger texture style feeling. This is more obvious in Frida Kahlo style images, and the Picasso style image qualities are quite similar and look very nice. The running time is in general longer as well.
In the end, The results of two images I like is shown below.
| Before Transfer | After Transfer | |
|---|---|---|
| Bear |  |
 |
| Flower |  |
 |