Assignment #5 - Cats Photo Editing

Project Overview

The project is split into three parts. Part1 is inverting the Generator, Part2 is Scribble to Image and Part3 is Stable Diffusion.

Part1 Inverting Generator

In the first part of the assignment, the task is to reconstruct an image from a latent code using a nonconvex optimization problem. The optimization problem is formulated as follows: given a real image $x$$x$, the goal is to find the optimal latent code $z^{\ast }$$z^*$ that minimizes the loss between the image generated by the generator $G(z)$$G(z)$ and the real image. The problem can be expressed mathematically as:

$z^{\ast }=\arg \underset{z}{min}\mathcal{L}(G(z),x)$$z^* = \arg\min_z \mathcal{L}(G(z), x)$

Part2 Scribble to Image

Part2 involves imposing constraints on image generation to ensure that specific attributes, such as color scribbles, are realistically adhered to in the final image. Initially, the task is approached in a general context before focusing on color scribble constraints.

The problem is to find an optimal latent code $z^{\ast }$$z^*$ that, when passed through a trained generator $G$$G$, minimizes the sum of losses between the generated image $G(z)$$G(z)$ and a set of predefined constraints. These constraints are expressed in terms of functions $f_i$$f_i$ equating to scalar values $v_i$$v_i$. The overall optimization problem is nonconvex and penalized, suggesting the addition of a penalty term to enforce the constraints:

$z^{\ast }=\arg \underset{z}{min}\sum _i|f_i(G(z))-v_i{|}_1$$z^* = \arg\min_z \sum_i |f_i(G(z)) - v_i|_1$

Given that the color scribble constraints are element-wise, the optimization problem can be simplified to operate on the constrained pixels only. The revised optimization problem under these constraints is:

$z^{\ast }=\arg \underset{z}{min}||M\ast G(z)-M\ast S|{|}_1$$z^* = \arg\min_z || M * G(z) - M * S ||_1$

where $\ast$$*$ denotes the Hadamard (element-wise) product, $M$$M$ is the mask indicating which pixels are constrained by the scribble, and $S$$S$ is the sketch image provided by the user. The $L_1$$L_1$ norm is used for the loss, focusing the optimization on meeting the scribble constraints.

Part3 Stable Diffusion

The part is to adapt Stable Diffusion, which typically generates images based on text prompts, to also incorporate an input image. This modified process, akin to SDEdit, uses the input image as a guide by diffusing it into noise and then denoising it to create a realistic image that aligns with the given text prompt. We utilize the DDPM sampling method with Classifier-free Diffusion Guidance for its implementation.

Part1 Inverting the Generator

We use fixed seed to be 0 and show the original generator's generation on one sample and our target image below.

Example

One potential generation results from the original sampled seeds:

Target image:

Ablation Study 1. various combinations of the losses including Lp loss, Preceptual loss and/or regularization loss that penalizes L2 norm of delta

We discuss different weights for L1 loss and preceptual loss and regulation loss while making generative model to be StyleGAN and latent space to be w+ (from mean) space. We only show results after 1000 iterations.

When fixing perc_loss_weight = 0.01 and regularization_loss_weight = 0, we discuss L1_loss_weight. When the L1 loss weight becomes higher, L1 loss incentivizes the network to generate images with sharper edges and more pronounced features. It pushed the network to fit the target image from the source image.

L1_loss_weight=0	L1_loss_weight=1	L1_loss_weight=10	L1_loss_weight=100

When fixing L1_loss_weight = 1 and regularization_loss_weight = 0, we discuss perceptual_loss. When the perceptual loss becomes larger, the generator would focus on the reference image more. However, instead of L1 loss that directly forces the model to generate perceptually correct image. However, generally perceptual loss does not encourage the model to mimic the reference image exactly.

perc_loss_weight=0.01	perc_loss_weight=0.1	perc_loss_weight=1	perc_loss_weight=10

When fixing L1_loss_weight = 0 and perceptual_loss = 0.01, we discuss L2 regularization loss on delta. When we apply the regularization loss on the image, it regularizes on the difference between the reference image and the original generated image. If the regularization loss weight is higher, the image would just be almost unchanged. If the regularization loss weight is lower, it would becomes more similar with the reference image.

reg_loss_weight=0	reg_loss_weight=0.01	reg_loss_weight=0.1	reg_loss_weight=1

Ablation Study2. different generative models including vanilla GAN, StyleGAN

We compare results of GAN and StyleGAN while making loss to be 10 L1 loss and 0.01 preceptual los and the latent space to be z space. We only show results after 1000 iterations. Compared with Vanilla GAN, StyleGAN provides a new style-based generator that better modify the style of the images at different levels of detail, allowing precise control over the synthesis process. Therefore, given the same z space, StyleGAN is able to generate more realistic image when having additional guidance.

Vanilla GAN	StyleGAN

Ablation Study3. different latent space (latent code in z space, w space, and w+ space)

We compare results for z space, w space, and w+ space while fixing the generative models to be StyleGAN and loss to be 10 L1 loss and 0.01 preceptual loss. We only show results after 1000 iterations. Generally, we found that z space is worse than w space and w space is worse than w+ space. Moreover, from_mean method is worse than the normal method.

For the first conclusion, we think that w+ space provides the most flexibility for style generation compared with w space and z space. It makes the model easier to learn the feature. Additionally, compared with random sampling from the w or w+ space, from_mean is more stable and avoid extremes in the distribution. Extremes might result in less realistic or lower quality images, as the generator is less likely to have trained effectively on such outliers.

z space	w space	w+ space	w space (from mean)	w space (from mean)

Speed Testing

Utilizing one A100 GPU, wallclock time for three runs of StyleGAN including 1000 iterations is 21.54s, 21.44s, 21.48s. Wallclock time for three runs of VanillaGAN including 1000 interations is 7.04s, 7.17s, 7.41s.

Best hyper-parameter setting

The best hyper-parameter setting is perceptual loss weight = 0.1, L1 loss weight = 10, and regularization weight = 0.1. This combination of hyper-parameter makes sure that the cat is not too similar with the target and not quite similar with the original generation, making it suitable to be utilized as style control.

Here is the generated image:

Part2 Scribble to Image

I show some examples of scribbling below. Typically, we can see that examples with sparser and denser sketches, the generated results are generally aligned with their typical part of color. However, some generation results do not have natural shapes. Even though generally the output can be considered as cat-shape, they suffer from issues like improper or unnatural color distribution and distortion.

When the sketch is super dense, the control is pretty strong and the cat is very unnatural. When the sketch is sparser, the shape becomes more natural while the color can be unnatural.

sketch	generation results

Part3 Stable Diffusion

Ablation Study1: 2 different amounts of noises added to the input

We discuss different timesteps ranging from 300, 500, 700, 900 and 999 and fix the strength to be 30. We found that when the timesteps becomes larger and larger, the generated image is actully more related to the text prompt and less related to the original image. It is mainly due to larger noise brings more freedom into the generation process.

Example1

Original Image:

Text Prompt:

Grumpy cat reimagined as a royal painting

Decoding Results:

Example2

Original Image:

Text Prompt:

a japanese anime character

Decoding Results:

Ablation Study2: 2 different classifier-free guidance strength values

We discuss different strengths ranging from 10, 30, 50, 70, and 90 and fix the stength to be 900. We found that when the strength becomes larger, the image becomes more related to the text prompt. However, when the prompt is too large like more than 80, the generation quality becomes worse and there appears some ambiguous points and features that we do not want.

Example1

Original Image:

Text Prompt:

Grumpy cat reimagined as a royal painting

Decoding Results:

Example2

Original Image:

Text Prompt:

a japanese anime character

Decoding Results: