1. Inverting a Pre-Trained Generator

The goal here is to find a latent variable z within the space that a generative model (like a GAN) uses, such that when z is input into the generator, the output closely matches a provided real image. This process is often called "inversion" because you are effectively working backwards from the image to the latent space, rather than the normal direction of generation.
Approach: Use a pre-trained generator. Define a loss function that measures the difference between the generated image and the target real image (e.g., pixel-wise mean squared error or perceptual similarity using a feature extractor like VGG). Optimize the latent variable z using gradient descent to minimize the loss, adjusting z until the generator’s output matches the target image as closely as possible.

Various combinations of the losses(Lp loss, Preceptual loss, regularization loss)

Different generative models including vanilla GAN, StyleGAN

Different latent space (latent code in z, w, w+ space)

Overall Comments

StyleGAN inversion typically yields higher-quality images than Vanilla GAN inversion, suggesting that the effectiveness of image projection correlates with the generative power of the GAN utilized. Inversions in StyleGAN's W space and W+ space tend to achieve greater resemblance to the original image, possibly due to these spaces allowing the generation of w-vectors that cannot be derived from any z-vector. Using perceptual loss generally enhances the results, with the most effective performance being noted in StyleGAN W space inversion using a perceptual loss of 0.01, and StyleGAN W+ space inversion with a perceptual loss of 1. In terms of processing time, Vanilla GAN inversion completes in 18 seconds, whereas inversions in StyleGAN Z space, W space, and W+ space each require about 29 seconds.

2. Image Generation from a Hand-Drawn Sketch

This part involves taking a simple, hand-drawn sketch and using it as a basis to generate a detailed and realistic image.
Approach: Employ a conditional generative model that can take sketches as input. Models like pix2pix or CycleGAN could be effective, as they are designed for image-to-image translation tasks. Train the model on a dataset where pairs of sketches and real images are available. The model learns to fill in details and textures to convert sketches into lifelike images.

Overall comments

The analysis shows that the images generated from the scribbles lack realism. This is largely due to the simplicity of most scribbles, which challenges the model's ability to accurately map these to real-life images. . Expanding the dataset might enable the model to produce higher quality images. Despite the lack of realism, the generated images do capture the basic colors of the original scribbles, with denser scribbles resulting in slightly better outcomes.

3. Generating Images with Stable Diffusion

Stable Diffusion is a powerful text-to-image model that can generate detailed images based on textual descriptions. This part involves manipulating an input image according to a provided text prompt.
Approach: Use the Stable Diffusion model, which combines a deep learning text encoder and a latent diffusion model. Provide the model with an initial image and a text prompt. The model then interprets the text to apply transformations or integrate new elements into the initial image. This could involve a process similar to "in-painting" where parts of the image are modified or overwritten based on the textual guidance, or more complex interactions between the text and the image content.

As I decrease each noise step to 0.5, the number of timesteps increased which results in less clarified image compared to the same timestep

As the strength setting increases, the resulting image becomes more aligned with the detailed prompt and shows finer details. An image generated at strength 15 closely matches the description compared to one at strength 5, which tends to be more abstract.

Bells & Whistles

Other Brilliant ideas that I came up with - ControlNet

I implemented controlnet which can add condition to stable diffusion model. Basically it works like this: It copys the weights of neural network blocks into a "locked" copy and a "trainable" copy. You can apply various control to the image using scribbles, canny images and segmentmaks and so on The "trainable" one learns your condition. The "locked" one preserves your model.