SLAPaper / ComfyUI_ELLA

ComfyUI Implementaion of ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

ComfyUI_ELLA

ComfyUI Implementaion of ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

Note

As per the ELLA developers / researchers, only the SD 1.5 checkpoint is released.

Installation

To install, simply navigate to custom_nodes, and inside that directory, do git clone https://github.com/ExponentialML/ComfyUI_ELLA.git

Models

These models must be placed in the corresponding directories under models.

Example: ComfyUI/models/ella/model_file

The directories should be created automatically. If they are not, please create them manually.

1. Place the ELLA Model under a new folder ella: https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors

2. Create a folder called t5_model. Navigate to that folder (you must be in that directory), and git clone https://huggingface.co/google/flan-t5-xl to download the t5 model.

3. If you don't wish to use git, you can dowload each indvididually file manually by creating a folder t5_model/flan-t5-xl, then download every file from here, although I recommend git as it's easier.

In summary, you should have the following model directory structure:

• ComfyUI/models/ella/ella-sd1.5-tsc-t5xl.safetensors
• ComfyUI/models/t5_model/flan-t5-xl/all_downloaded_t5_models

Usage

Workflow

To get started quickly, a workflow is provided in the workflow directory.

ELLA Loader

• ella_model: The path to the ella checkpoint file.
• t5_model: The path to the t5 model folder.

ELLA Text Encode

• ella: The loaded model using the ELLA Loader.
• text: Conditioning prompt. All weighting and such should be 1:1 with all condiioning nodes.
• sigma: The required sigma for the prompt. It must be the same as the KSampler settings. Without the workflow, initially this will be a float. You can simply right click the node, convert sigma to input, then use the Get Sigma node.

Support

All conditioning nodes should be supported, as well as prompt weighting and ControlNet.

Attribution

Thanks to the following for open sourcing. Please follow their respective licensing.

• ELLA: https://github.com/TencentQQGYLab/ELLA

• Get Sigma Via BNK: https://github.com/BlenderNeko/ComfyUI_Noise

• Diffusers (borrowed timestep modules): https://github.com/huggingface/diffusers