Assignment 4

Content Reconstruction

Optimizing Different Layers

Original Image

Conv 1

Conv 4

Conv 5

Conv 7

It is evident that putting our content loss earlier seems to retain most of the content image, with conv_1 and conv_4 having very good results.

My results (conv_1)

Noise for Falling Water

Falling Water Reconstructed

Noise for Phipps

Phipps Reconstructed

We can see that the images are basically perfectly reconstructed, with no noticeable differences.

Texture Synthesis

Optimizing Different Layers

Original Texture

Conv 1, Conv 3, Conv 5

Conv 1, Conv 2, Conv 3, Conv 4, Conv 5

Conv 3, Conv 4, Conv 5, Conv 6, Conv 7

Conv 8, Conv 9, Conv 10, Conv 11, Conv 12

Again, it seems like placing the style loss on earlier conv layers seems to synthesize the best textures, with later layers making textures look quite noisy.

My results

Input Noise

Synthesized Scream Texture (conv_1, conv_2, conv_3, conv_4, conv_5)

Input Noise

Synthesized Texture

In my opinion, the synthesized textures retained the information quite well!

Style Transfer

Hyperparamater Tuning

Style Weight: 10⁴

Style Weight: 10⁵

Style Weight: 10⁶

Style Weight: 10⁷

Judging by the results above, I preferred to use a style weight of 10⁶ since it gives a good blend of retaining the content image, but really emphasizing the style image. I also found that tuning the learning rate was required. The default LR of 1 worked well for some images, but often times it was too high, so I found that a value between 0.05 and 0.1 worked best.

Results

Noise vs Content Image

From Noise

From Content Image (ignore the title of the image)

One big change I noticed when initializing with the content image is that the loss starts off way lower, however the runtime for both was approximately the same, and the results seem similar, but I think that the noise initialized image has more noticable artifacts that don't appear in the original image. For example, the sky in the left image seems to be more fragmented between blue and white-ish patches, while the image on the right has this more consistent.