For the final project, we were allowed to choose our own topics. I chose to work on a pipeline to place myself in interesting new locations and then stylizing the images. As an aspiring machine learning student in the field of computer vision, it is almost a cliché to have a stylized profile picture, but hey, it looks cool.
The initial plan was to have a two part pipeline: one to insert myself into a new scene, and the other to stylize the image. I planned to use diffusion model for conditional text-to-image generation, so the only important details is figuring out where to insert images of myself into the process as well as how to insert the style. To make these design choices, I came up with several approaches inspired by our class lectures. For content-based portion of the pipeline, I considered the following:
For the style-based portion of the pipeline, I considered the following:
To determine the best result for each design choice, I performed several experiments as well as investigated literature and existing replications online. In addition to the actual model implementation, I also performed experiments with training datasets and hyperparameters to see what worked best for each model.
Textual inversion is reportedly a much cheaper version of fine-tuning that I thought worked very intuitively. A new embedding learns the conditional latent space to find the latent that minimizes the distance between the training images and a conditioned generated image.
I implemented this model and found that it was fairly slow to run, and the results weren't very good. To get a diffusion model to fit onto a T4 GPU, which is what is available through AWS, I used 16 floating point quantization. I also leveraged FAIR's xFormers for memory efficient attention. These augmentations were enough to get the model to fit onto the T4.
Experiment 6 tried to increase the number of timesteps to improve the quality of the image, but the model doesn't appear to be trained for that, and I got a completely black screen as the final output. Experiments 7 and 8 tested the impact of guidance and prompts, but the quality of output did not improve.
The last experiment with textual inversion tried reducing batch size from 4 to 2. The results were the strongest yet, but it was clear that textual inversion was not powerful enough for the task at hand.
I never implemented extended textual inversion, but I did mention trying this in my initial plan. I did not implement this for two reasons.
I decided it would be best to just move onto DreamBooth.
Traditionally, DreamBooth has a clever trick to prevent overfitting to an instance of a class. They refer to this as the "prior preservation". However, for my specific use case, I only care about myself, so preserving the class while training the instance is not important to me.
Starting where I left off with textual inversion, I used the same dataset. Luckily, the field of image generation is fairly popular with a strong community of hobbyists that have heuristically developed a lot of best practices for fine-tuning. I read online that the best practice for DreamBooth is for full finetuning to use a lower learning rate. and to have about 10x the number of training iterations as the number of images in the dataset.
Never doubt Reddit.
Now that I have some decent results, I can start to think of improving the model.
Experiment 15 had pretty strong results, so have some more:
Same with Experiment 17, which used new training prompts.
For stylization, I've already successfully implemented a quantized diffusion model that fits onto the T4 I'm using on AWS, so I figured I might as well try to use the same model for stylization. For that reason, I started with SDEdit.
As we have seen in the homework, SDEdit works by noising an image to the point (e.g. halfway through the noising process) where only lower frequency information is retained. From there, we can denoise with a prompt that adds style at the high frequency space. There are limitations to this approach, for instance, we cannot preserve high frequency information during the noising process, and we cannot introduce low frequency style details into the image.
While denoising, there is an additional hyperparameter that I need to experiment with during inference. In particular, I don't know how far to noise, so I have to try at different values to see what works best for each image.
I happen to like Borderlands, so I went online, found around 138 Borderlands images, then performed random cropping to get 212 images (larger images were randomly cropped more times through an empirical formula I developed).
Naruto was a failure case, but I will show my experiments here. I first scraped 104 images from the first 63 episodes of Naruto. Then I used the same random cropping preprocessing step to produce 450 images.
In particular, I dislike how I keep getting whiskers on my cheeks (a characteristic of the main character, whose name is also Naruto). I suspect that the model has some prior weights that associate "Naruto-style artwork" to the character rather than the TV show, which was my intention.
I next tried to fix the issue of generating a more accurate Naruto-style image. Instead of random cropping, which tended to only have partial images of faces, I used resizing to generate the training dataset of 104 images.
The face lines are reduced, but the images aren't very good. For a hail mary, I focused on only one character called Sasuke, then the prompt would be the right style and not involve "Naruto" in the prompt, which I suspected was a confounding variable. The results were a nightmare that should not be shown to the general public. I decided to move onto a new style.
Conveniently, Ghibli provides a lot of high quality images on their website. I downloaded 250 images from 5 separate movies, then resized them as needed.
