KaihuaTang / Local-Disco-Diffusion-v5.2.jupyterNote

A custom Disco Diffusion v5.2 that runs on local GPUS.

Local-Disco-Diffusion-v5.2

A custom Disco Diffusion v5.2 that runs on local GPUS.

Contents

1. Background
2. Install the Requirements
3. Let's Disco
4. Useful Tutorials
5. Two Versions

Background

Fig. 1: An image generated by Disco Diffusion v5.2 with the default prompt "A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade, Trending on artstation.".

Disco Diffusion (DD) is an open-source project that generates high-quality images from the given text description, which is also known as prompt. The technique used by DD is called CLIP-guided Diffusion, where the CLIP is an multi-modal pretrained model that connecting text and images, and diffusion model is a kind of generative model that iteratively update the image from Gaussian noise to a cleaner and more detailed version of the original one (Fig. 2). Combining the above two techniques, DD is then created by Somnai, augmented by Gandamu, and building on the work of RiversHaveWings, nshepperd, and many other collaborators. A detailed introduction of Disco Diffusion can be found at URL. A demo of Disco Diffusion can also be freely obtained, modified, and executed on Colab.

However, free users have plenty of limitations and restraints on Google Colab. For those researchers, artists, or engineers who have their own high-performance GPUs, we provide a custom version of Disco Diffusion v5.2 project that runs on local machines, so we can better realize its full potentials.

Fig. 2: an example of images generated at intermediate steps of the diffusion process using the prompt: "A girl and a boy meet each other in a forest under the stars from a Hayao Miyazaki animation."

Install the Requirement

###################################
###  Step by Step Installation   ##
###################################

# 1. create and activate conda environment
conda create -n disco_diffusion pip python=3.7
source activate disco_diffusion

# 2. install jupyter notebook
conda install jupyter notebook

# 3. generate jupyter config
jupyter notebook --generate-config

# 4. open "~/.jupyter/jupyter_notebook_config.py" and add the following three lines
c.NotebookApp.ip = 'YOUR_IP'
c.NotebookApp.open_browser=True
c.NotebookApp.password_required=True

# 5. set up the password of your jupyter notebook
jupyter notebook password

# 6. install conda packages
conda install pandas regex matplotlib

# 7. install pip packages
pip install opencv-python lpips datetime timm ftfy einops pytorch-lightning omegaconf

Let's Disco

After install all the requirements, we can start to run the disco diffusion project. We can open the jupyter notebook on your local device by the following command:

# open jupyter notebook
jupyter notebook

then we can run the Local_Disco_Diffusion_v5_2.ipynb to generate your amazing images!

The text prompts can be gound at Section 3 in the notebook file. You can type multiple sentences to describe the image with the importance weight ":Number" at the end of each sentences like ["a realistic image of a ginger cat:2", "a beautiful kitchen:1"].

Useful Tutorials

More detailed instructions about the parameters of Disco Diffusion project can be found at Zippy's Disco Diffusion Cheatsheet v0.3. To study the effect of different artists, we can see a summary of their painting styles at Disco Diffusion 70+ Artist Studies. Similarly, there are some popular Disco Diffusion Modifiers that can be added as the end of the prompt for certain artistic effects.

1. Zippy's Disco Diffusion Cheatsheet v0.3
2. Disco Diffusion 70+ Artist Studies
3. Disco Diffusion Modifiers

An example of prompt created by following the above tutorials:

"a beautiful painting of a deer with tentacles as its mouth walking across the forest by Karol Bak, trending on artstation, Lovecraftian."

Fig. 3: Prompt: "a beautiful painting of a deer with tentacles as its mouth walking across the forest by Karol Bak, trending on artstation, Lovecraftian."

Two Versions

There are two versions of jupyter notebook files for the Disco Diffusion project.

1. The file Local_Disco_Diffusion_original_v5_2.ipynb only has the necessary modifications with minimum changes. All functions are still kept in one single file just like the Colab version.

2. The file Local_Disco_Diffusion_v5_2.ipynb puts all the functions in separate utils files, so it only contains the core function and all the parameters.