Assignment #5 - Cats Photo Editing

Yifei Liu

Introduction

For this assignment, we had some fun with editing cats by experimented with different image manipulation techniques. First, we invert a pre-trained generator to amtch a real image with its latent counterpart. Next, we transformed sketches into realistic images. Lastly, we employed stable diffusion to generate images from both visual and text prompts.

Part 1: Inverting the Generator

We first perform the task of reconstructing target image from its latent representation. We formulated a nonconvex optimization problem as follows, with the premise that a well-trained generator's output manifold can be a close approximation to that of natural images.

I experimented with different Lp losses in combination of perceptual loss and uses LBFGS optimization. I tried different generative models: vanilla GAN and StyleGAN, and three different latent space: z space, w space, and w+ space. Results below are image projections after 1000 optimization steps.

The caption of images mark the parameters as [imgName]_[model]_[latentSpace]_[percentualLossWeight] and the difference is highlighted in purple.

Results

Vanilla GAN

Style GAN

Ablation

StyleGAN has better reconstruction results than vanilla GAN with advanced architecture that uses adaptive isntance normalization and is tailored for style transfer.

L1 and L2 loss with z latent space lead to similar results, while L1 work a lot better with w latent space (see first row). W space emphasizes the overall "style" or "feel" of the image, such as textures and colors. L1 loss aligns with this objective by encouraging the preservation of broader image features without being disproportionately affected by extreme differences in any specific pixels.

W+ and W latent space are comparable with w+ slightly blurrier than w and has less variation in the reconstruction because it has risk of overfitting with the freedom of unique adjustment per layer.

Percetual loss weight doesn’t affect the result too notably but is expected that a lower weight will make the model priortize Lp loss and tend to generate images closely matching the target images in terms of pixel-wise error but might not capture high-level perceptual features, which can explain why the result with weight 0 has slightly less features.

Overall, I think StyleGAN with a perceptual loss weight of 0.01, l1 loss, w latent space yield the best result.

On a RTX 4090 GPU, the runtime of optimization for 1000 steps are as followed:

• stylegan w: 15.02 sec

• stylegan w+: 14.37

• stylegan z: 15.04

• vanilla z: 4.92

Additional Results

Part 2: Scribble to Image

In this part, we focused on infusing user-defined constraints into image generation process. By introducing color scribble constraints into a soft-constrained optimization framework, we guide the GAN model in generating realistic images while adhering to user inputs (sketches). The optimization is:

where * is the Hadamard product, M is mask, and S is the sketch. We minimize the difference between the generated image and the hand-drawn color scribbles, considering each pixel's value within a designated mask.

I chose StyleGAN as the model and w+ latent space with percetual loss weight of 0.01. w+ space perform lot better than w space in this task with its ability t do fine-grained control over generated image features at different layers.

Results

I tried different amount of user-input, from sparser to denser sketches.

Sketches

Mask

Generated

When using sparser sketches with just the outlines of cat, the generative model has more freedom to infer the missing details and textures, which lead to better results above as the model fill in the gap from learned knowldge of relistic cats. It still suffer from photorealistic rendering though and requires more sophisticated model trained on richer dataset that captured wide range of textures, colors, and shapes.

Stable Diffusion

Lastly, in exploration into guided image synthesis, we expanded the capability of stable diffusion, to integrate an input image alongside the text as guides through the diffusion process. This approach, inspired by SDEdit, leverages the input image to direct the generation process, which is then refined through DDPM sampling and Classifier-free Diffusion Guidance.

The approach is as followed:

Results

I experimented with different amounts of noises added to the input (by varying the timestep of DDPM sampling) and differnet classifier-free-guidance strength values.

Prompt: “Grumpy cat reimagined as a royal painting”

As we can see above, increasing the timestep, thereby the added noise, make the result align more closely to the text prompt at the expense of adhering less to the user sketch. A higher guidance strength pushes the model to generate images with more details and follow text prompt more closely but also has a risk of making it favor certain features or patterns it has strongly associated with the text.

I chose timestep=700 and strength=15 and tried on some of my own drawings.

Prompt

“Front of fancy museum in alfred josephy casson's style”

“Night forest with trees and moon in Alexandre Benois style”

“Beautiful ocean scene with a boat in style of Chiho Aoshima”

Input Image

Generated