1. Overview
1.1 Introduction
Image editing has transcended from simple manipulations to complex transformations that leverage the manifold of natural images. With GANs and diffusion models, the ability to generate, modify, and invert images has reached new heights. GANs, comprising two neural networks that compete against each other, enable the generation of realistic images by learning the distribution of training data. The generator creates images, while the discriminator evaluates them, iteratively improving the quality of synthetic images. Meanwhile, diffusion models, which reverse a process that adds noise to data, allow us to craft images by starting from a state of pure noise and gradually denoising to obtain a coherent structure.
The assignment is structured into three main parts. Initially, Part 1 lays the groundwork by focusing on the inversion of a pre-trained generator model to find latent variables that can accurately reconstruct given real images. Following this, Part 2 is about bringing a hand-drawn sketch to life by generating an image that aligns with the given sketch, emphasizing the creative potential of GANs in art and design. Finally, Part 3 combines both visual and textual input by using diffusion model, thereby enabling a more controlled and descriptive creation that is conditioned on both the textual prompt and the guiding input image.
2. Implementation
Part 1: Inverting the Generator
The application of varying loss function combinations in image reconstruction elucidated their distinct impact on the output. While keeping the Lp loss weight constant, modifying the perceptual loss influenced texture fidelity and image clarity, with a larger weight enhancing structural features. The regularization loss, theoretically a means to ensure smooth gradients, manifested a trade-off: higher weights introduced unwanted artifacts indicative of over-regularization, whereas too low a weight failed to constrain the noise, leading to blur. The balance between detail preservation and noise suppression was critical.
In evaluating generative models, the foundational difference between vanilla GANs and StyleGAN became apparent. Vanilla GANs, due to their simpler architecture, were less adept at capturing intricacies, resulting in more abstract reconstructions. StyleGAN, however, leverages a more complex architecture that better captures and reproduces the nuances of the input image, resulting in reconstructions with greater fidelity, which aligns with its theoretical framework designed for detailed synthesis.
Delving into StyleGAN's latent spaces, each displayed unique characteristics attributable to their dimensional properties. The Z space, directly tied to random noise, produced some artifacts, potentially due to its direct encoding method. The W space, offering a transformed and more disentangled representation, showed improved quality, reducing artifacts. The W+ space, an augmented version of W, provided even finer control over features at the cost of some blurriness, which could be a result of overfitting in high-dimensional space.
The algorithm takes on average 30 seconds to run for 64 resolution, including the time to initiate the model. At first, it would took several minutes, but by fixing the LBFGS optimizer clusure function, it is relatively fast at the moment for small pictures.
Part 2: Scribble to Image
The outputs generated from the sketches of cats using the StyleGAN model reveal interesting insights into the capabilities of the W and W+ latent spaces. For sparser sketches, the W latent space has filled in missing details, utilizing the model's inherent knowledge to produce a coherent image. This suggests that W is better for interpreting incomplete information and still generating a recognizable cat face. But the results are lacking in diversity. Conversely, the W+ space, which provides more nuanced control over the generated image, has adhered more closely to denser sketches and has accurately rendered the additional information provided by color.
The denser the sketch and the more precise the coloration, the more detailed and accurate the W+ outputs are. This fidelity can be attributed to the W+ space's ability to manipulate individual features across multiple layers of the neural network, making it ideal for sketches that provide a rich source of information. In the W+ column, we notice the colors and shapes are better matched to the input image, reflecting the model's capacity to leverage detailed inputs to produce high-quality outputs.
Part 3: Stable Diffusion
Stable Diffusion synthesizes images by iteratively refining noise into a coherent picture. Unlike GANs, which involve a discriminator and generator in an adversarial process, Stable Diffusion starts with a guide image and a corresponding textual prompt. The guide serves as an anchor, dictating the initial direction of the image synthesis, which is then augmented by the text to add specific details or themes. The process leverages a pre-trained diffusion model that initially transforms the guide into a noisy version by adding sampled noise. This noisy version is then gradually denoised through a series of timesteps, during which classifier-free diffusion guidance comes into play. This guidance adjusts the denoising steps to ensure that the resulting image adheres to the text prompt while retaining the essence of the guide image. It's a balance between following the guide (faithfulness to the initial image) and fulfilling the text prompt (realism and relevance to the text).
Some experiment results:
A comparison of generated images using (1) 2 different amounts of noises added to the input and (2) 2 different classifier-free guidance strength values.
The synthesis outputs from two distinct input images demonstrate how varying noise levels and guidance strengths impact the generation process. The first comparison involves adjusting the noise levels by altering the number of diffusion steps. With fewer steps, the output closely mirrors the original sketch, maintaining high faithfulness. Increasing the steps introduces more noise, which, while contributing to a more realistic texture, diverges from the exact features of the input image.
The second variable, classifier-free guidance strength, directly influences the adherence of the generated image to the input guide. A lower guidance strength retains the structural essence of the guide, resulting in images that prioritize faithfulness. In contrast, a higher guidance strength pushes the model to emphasize realism, creating images that, although more detailed and textured, may exhibit less similarity to the input. But too much strength seems to create artifacts. This parameter essentially controls the creative freedom of the model, allowing it to either respect the guide or explore beyond it.
Bells & Whistles
1. Interpolate between two latent codes in the GAN model, and generate an image sequence (2pt)
2. Develop a cool user interface and record a UI demo (4 pts). Write a cool front end for your optimization backend.
For the front end of the optimization backend, I employed a combination of Django and React. Django serves as a robust and secure backend framework, while React allows for the creation of a dynamic and responsive user interface. I took this bell and whistle to brush up a bit my web app building skills. One potential problem for actually turning it to an app could be the processing time, 500 steps takes one and a half minute to finish and can be very dicouraging.
3. Implement other Diffusion model’s few-step samplers (eg., DDIM) for the stable diffusion part, and compare its results. (4 pts)
DDPM is a stochastic process that transforms an image into Gaussian noise over a series of timesteps and then reverses this process to generate an image. This reverse process is stochastic, meaning it incorporates randomness at each step, which can lead to diverse outputs. DDIM, on the other hand, is a non-stochastic variant that allows for faster sampling by taking deterministic steps. This means that given the same initial conditions and parameters, DDIM will always produce the same output, unlike DDPM, which could yield different results under the same conditions due to its inherent randomness.
Theoretically, DDIM (Denoising Diffusion Implicit Models) should maintain a close relationship to the input image, given its deterministic nature in the denoising process. However, the actual results deviate from this expectation. When using the same sigma_t as DDPM, which is known for its stochasticity, the DDIM-generated images differ significantly from the original guide. This divergence could potentially be attributed to the influence of the text prompt, which might be steering the generation process away from the input image, favoring the aspects dictated by the text. In scenarios where sigma_t is set as defined in the DDIM paper—a larger value—the results adhere more closely to the guide image. However, they also display a certain murkiness. This lack of clarity in the output could be due to the larger sigma_t values creating a smoothing effect that dampens high-frequency details, resulting in outputs that, while resembling the guide, are more diffuse and less sharp. In DDIM, larger sigma_t values mean that the denoising path at each step is weighted more heavily towards the model's learned priors, which can sometimes generalize the details and produce "murkier" images. The model's learned priors, influenced by the textual prompt, may not always align perfectly with the input image's details, leading to this murkiness when attempting to reconcile the two sources of information—the image guide and the text prompt.