This is an efficient implementation of Stable-Diffusion-XL. I make the following improvements:
- Reconstruct the architecture of UNet. This UNet implementation is faster than others (Diffusers, Fooocus, etc.). If you are interested in this implementation, please see
FastSDXL/BlockUNet.py. The source code of this UNet is short and easy to understand. You can also use this component in your own projects. - Use a trainable scheduler named OLSS. The implementation of OLSS is based on this project. I find this scheduler can improve the quality of generated images with given steps but we need to train it first (the training process requires a few minutes). To synthesize images with a specific style, OLSS is a good choice.
The code is headless. I developed this project based on diffusers==0.21.3. If you find it cannot run with another version of diffusers, please open an issue and tell me.
pip install diffusers safetensors torch gradio
To launch a webui without OLSS scheduler, please run the following command.
python launch_without_olss.py
To train an OLSS scheduler, please see run_olss.py for more details.
I trained an OLSS scheduler with the style of "Slightly Cinematic". You can use it by running the following command.
python launch_with_olss.py
I tested my code using NVidia 3060 laptop (6G, 85W). The resolution is 1024*1024, and the model is converted to float16 format.
- Diffusers: CUDA out of memory
- Fooocus: 1.78s/it
- FastSDXL (ours): 1.17s/it
Here are some examples of OLSS.
Prompt template: cinematic still {prompt}. emotional, harmonious, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy Negative prompt: anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured
- a young girl, black hair, white clothes, in a garden, the background is red and white flowers
In the image generated by OLSS, the details of flowers on the right are more realistic.
| DDIM | DPM | OLSS |
|---|---|---|
 |
 |
 |
- a forest in spring, birds
More birds.
| DDIM | DPM | OLSS |
|---|---|---|
 |
 |
 |
- an orange cat and a pink ball on a white sofa
In this example, the image generated by OLSS is completely different with others.
| DDIM | DPM | OLSS |
|---|---|---|
 |
 |
 |
- a robot, with blue lightsaber, in a city
Sometimes OLSS can modify the composition and fix the composition error.
| DDIM | DPM | OLSS |
|---|---|---|
 |
 |
 |