Assignment #4 - Neural Style Transfer

In this project, we implement neural style transfer which can transfer specific contents to resemble a certain artistic style. For the first part, we will optimize random noise in content space to get familiar with optimizing pixels to content losses. For the second part, we will optimize random noise to match the textures of the style image. Then for the final part, we combine the first two steps and optimize some input image (initialized from random noise or content image) to match the artistic style of the style reference image.

 

For Bells & Whistles, I will first stylize the Poisson blended images from the previous homework. And then I will try using image editing methods for style transformation and compare the results.

In this part, we will use content loss to reconstruct the content image from random noise.

I am using the fallingwater.png as content image and trying to reconstruct the image using content loss only. The content loss measures the content distance between two images at some individual layer in the VGG19 feature layers. We first sample random noise as the original input image, and then optimize this input image in pixel space using the content loss. The content loss measures the MSE distance of the features after certain convolutional layers for the input image and the content image. By backpropagating through the loss, we are optimizing the input image to be more similar to the content image. To implement this loss, I insert a content_loss layer after the convolutional layers specified below.

 

Since the style image and content image should be of the same size, I am cropping all images to be size (512, 512).

 

From the results we can observe that when the layer is the more shallow layers such as Conv2, the reconstruction is almost identical to the original image. When the layer is deeper, the reconstructed image have more noise.

 

I choose Conv4 as my favorite layer because it preserve decent content information and also removes some textures of the content image, which will be better for further style transfer.

With content loss inserted after Conv4, I use fallingwater.png and dancing.jpg as two content images and select two different seed to generate the initial noise as two input images, and optimize them with content loss.

For using random seed 42 and 17, the differences are minor for both content images and we can observe that both seeds preserve the structure of the content image very well. This demonstrate the model's robustness.

In this part, we will implement style_loss to perform texture synthesis from random noise. We initialize the input image to be some random noise, and take a style image as reference where we want the input image to synthesize the textures of the style image. The style image and the input image is both fed through the VGG19 feature extrator, and after some specific convolutional layers, we measure the difference of the extracted features of the style image and the input image using Gram matrix. Then Gram matrix is a square matrix obtained by multiplying the featrure matrixx with its transpose, and can capture the statistical relationship between the two extracted features in terms of texture and pattern.

 

To implement the style_loss, I calculate the Gram matrix as described and calculate the MSE distance between that of the input image and the style image. The style_loss is inserted after each of the specified convolutional layers below.

For the configurations, I am only using the style loss with a weight of 100000, and a default num_steps=300. For random noise generation, I am using seed=42 to generate the noise as input image.

 

For the style image, I am using the frida_kahlo.jpeg and trying to learn the textures represented in this image.

 

 

From the results, we can see that the shallow layers will lead to blurry textures that don't have enough resemblence to the reference style image. However, when the layer goes deeper, the noise level will also be too high to observe the textures. Therefore, I am selecting Conv1-Conv5 as the layers after which I insert the style_loss, and as shown in the table, this can lead to synthesized textures that well preserve the original texture information.

For generating different random noises, I am using seed=42 and seed=17 to get two input images and optimize them only with style loss for the style image picasso.jpg and frida_kahlo.jpeg.

We can observe that for different seeds, the synthesized textures are pretty similar with each other, which shows that our style_loss is robust enough to learn texture information from style images.

I select two samples generate in Assignment 2 where we use Poisson blending to fuse two images together, and perform style transfer. The results are shown in the following table.

Recent years, we have a lot of generation models that are trained with huge amounts of data, and can generate stylized images given text prompts. Based on these generation models, there are also image editing models that can condition the generated images on some input image and text prompt.

 

InstructPix2pix is one of the most recent work that can perform image editing in an end-to-end fashion without any fine-tuning. Therefore, I am testing the InstructPix2pix model's performance on the style transfer task.

 

For the prompt for InstructPix2pix, I am using "Make it a picasso painting." and "Turn this into the style of starry night of Van Gogh."

From the results, we can observe that InstructPix2pix have troubles keeping the original structure of the content image, but can apply the style changes most of the time. While the style transfer method we implemented always keep the original structure and details of the content image, it sometimes cannot make obvious or significant style transfer. Moreover, one of the advantage of InstructPix2pix is that, we don't actually need a reference image to perform the style transfer. Although text descriptions have ambiguitiy, it is much easier to collect.

 

For computation statistics, the inference time for InstructPix2pix on A4500 is 15-16 seconds on average, which is about the same as running style transfer with 10 optimization steps. But the memory requirement for InstructPix2pix is about 18GB, while the requirement for style transfer is much smaller.

 

Overall, the end-to-end models usually have less guarantee on the output image structures, but generally are more creative and more prone to make huge changes. The style_loss and content_loss guided style transfer have more guarantee in the generated image's resemblence to the content image, and generally will pick up exactly the textural details of the provided style image.