StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Or Patashnik*, Zongze Wu*, Eli Shechtman, Daniel Cohen-Or, Dani Lischinski
*Equal contribution, ordered alphabetically
https://arxiv.org/abs/2103.17249Abstract: Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable textbased manipulation. Finally, we present a method for mapping a text prompts to input-agnostic directions in StyleGAN’s style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.
Official Implementation of StyleCLIP, a method to manipulate images using a driving text. Our method uses the generative power of a pretrained StyleGAN generator, and the visual-language power of CLIP. In the paper we present three methods:
- Latent vector optimization.
- Latent mapper, trained to manipulate latent vectors according to a specific text description.
- Global directions in the StyleSpace.
Currently the repository contains the code for the optimization only. The code for the latent mapper and for the global directions will be released soon - stay tuned!
31/3/2021 Upload paper to arxiv, and video to YouTube
14/2/2021 Initial version
In the following, we show some results obtained with our methods. All images are real, and were inverted into the StyleGAN's latent space using e4e. The driving text that was used for each edit appears below or above each image.
The code relies on the official implementation of CLIP, and the Rosinality pytorch implementation of StyleGAN2. Some parts of the StyleGAN implementation were modified, so that the whole implementation is native pytorch.
- Anaconda
- Pretrained StyleGAN2 generator (can be downloaded from here)
In addition, run the following commands:
conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=<CUDA_VERSION>
pip install ftfy regex tqdm
pip install git+https://github.com/openai/CLIP.gitGiven a textual description, one can both edit a given image, or generate a random image that best fits to the description.
Both operations can be done through the main.py script, or the notebook.
To edit an image set --mode=edit. Editing can be done on both provided latent vector, and on a random latent vector from StyleGAN's latent space.
It is recommended to adjust the --l2_lambda according to the desired edit.
To generate a free-style image set --mode=free_generation.
The global directions we find for editing are direction in the S Space, which was introduced and analyzed in StyleSpace (Wu et al).
To edit real images, we inverted them to the StyleGAN's latent space using e4e (Tov et al.).
If you use this code for your research, please cite our paper:
@misc{patashnik2021styleclip,
title={StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery},
author={Or Patashnik and Zongze Wu and Eli Shechtman and Daniel Cohen-Or and Dani Lischinski},
year={2021},
eprint={2103.17249},
archivePrefix={arXiv},
primaryClass={cs.CV}
}