AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.
- Clone this repo into
custom_nodes
folder. - Download motion modules and put them under
comfyui-animatediff/models/
.
- Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk
- Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff
- AnimateDiff v2 mm_sd_v15_v2.ckpt
Download motion LoRAs and put them under comfyui-animatediff/loras/
folder.
Note: LoRAs only work with AnimateDiff v2 mm_sd_v15_v2.ckpt module.
![image](https://private-user-images.githubusercontent.com/133728487/270393962-7a9f62f7-702e-48a4-934c-bbfe1e23aff2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjcwMzkzOTYyLTdhOWY2MmY3LTcwMmUtNDhhNC05MzRjLWJiZmUxZTIzYWZmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kNTM2MGI0NmRjZWJkYzQ0N2Y2NjgyMjg0MGI0ZTc2YjQyMWI4ZjFjMzQ5MzQ3Y2NiZmU3OTk5OTMxY2FhMDg5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.wQ52XK6__6j_O-Y6euAkKUdpyASbuQVIGMDCtcTcCEU)
Workflow: lora.json
Samples:
![]() |
![]() |
![]() |
![]() |
The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the SlidingWindowOptions
node. See the sample workflow bellow.
![image](https://private-user-images.githubusercontent.com/133728487/268482819-9d756d01-ea45-4d1c-8e48-56f2725c7ca1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODE5LTlkNzU2ZDAxLWVhNDUtNGQxYy04ZTQ4LTU2ZjI3MjVjN2NhMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lY2RlOWQ0ZjlkMWNkZmZmNDJmY2ZiYzY3Y2JiMWIyNTljZTI4MmJlMGFkZGZjMGZlNTZkNDA2ZDNhNGE5NTI5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.6CCGYj7IHVnnazrsiNg7GXrLbGTaXYPBiIzkbsjC8-M)
- Mostly the same with
KSampler
motion_module
: useAnimateDiffLoader
to load the motion moduleinject_method
: should left defaultframe_number
: animation lengthlatent_image
: You can pass anEmptyLatentImage
sliding_window_opts
: custom sliding window options
![image](https://private-user-images.githubusercontent.com/133728487/269430269-a352195d-f40c-494d-bd3d-30ee88174b88.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMwMjY5LWEzNTIxOTVkLWY0MGMtNDk0ZC1iZDNkLTMwZWU4ODE3NGI4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jNWE3OTFjOTJmNWM3Mjc4MWUzZTY0YzQ4NWI0OTM4YWUyOTBmNDA1YWM1MWQ2ZWZjYmNhNDFkMWQzMzBkMzE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.-KPM9MwyYwltPApAfQnVQ3BzTppYRXlxPbR6vva6VAs)
- Combine GIF frames and produce the GIF image
frame_rate
: number of frame per secondloop_count
: use 0 for infinite loopsave_image
: should GIF be saved to diskformat
: supportsimage/gif
,image/webp
(better compression),video/webm
,video/h264-mp4
,video/h265-mp4
. To use video formats, you'll need ffmpeg installed and available inPATH
![image](https://private-user-images.githubusercontent.com/133728487/268482889-381c5acc-06ef-43da-ada0-3dc76f37a3e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODg5LTM4MWM1YWNjLTA2ZWYtNDNkYS1hZGEwLTNkYzc2ZjM3YTNlNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04YTMwY2MzNmRmZDBmNmNjMjkzYTEyMmFhNmIwMjE2NTI2YzBhYmY5ZmViMzY3NDQ0OTVkNjY5OGM5MWM3ZTU2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.neEI1Eycu32eowho7YRx7G8KW_zyC1aZi1F0x72rYI0)
Custom sliding window options
context_length
: number of frame per window. Use 16 to get the best results. Reduce it if you have low VRAM.context_stride
:- 1: sampling every frame
- 2: sampling every frame then every second frame
- 3: sampling every frame then every second frame then every third frames
- ...
context_overlap
: overlap frames between each window sliceclosed_loop
: make the GIF a closed loop, will add more sampling step
![image](https://private-user-images.githubusercontent.com/133728487/269431763-6679a8dd-bf96-419f-8934-ea2b046dd23c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMxNzYzLTY2NzlhOGRkLWJmOTYtNDE5Zi04OTM0LWVhMmIwNDZkZDIzYy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05Yzc5MmE3OGY2NTBhYTUzYjZlNTY2ZDZlNWFhMGQxZmFjNDk2OTE2YWQzMzljY2EwYzdkY2FlZDI4OGM4NDM5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oBTjeHxkH-OsNOs4-RLjQhlSBVS4vbD-qIbUxj0osD0)
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
frame_start
: Skip some begining frames and start atframe_start
frame_limit
: Only takeframe_limit
frames
![image](https://private-user-images.githubusercontent.com/133728487/269429598-684176d5-6369-4a27-9f33-e721e0fe1876.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDI5NTk4LTY4NDE3NmQ1LTYzNjktNGEyNy05ZjMzLWU3MjFlMGZlMTg3Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hODA3NTQ3ZWFmZTg3NTIwNjZjZmQyYjhmNDBiYTI4ZWFkYjQ5YThkMjVjMDU0MmUxYjBlN2EzYWFjY2Y3NzQ2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.uGtaE5DgZWNxTfJG_0dohe5oWqlQrOvM52b2oze5e38)
![image](https://private-user-images.githubusercontent.com/133728487/268311503-b7164539-bc58-4ef9-b178-d914e833805e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4MzExNTAzLWI3MTY0NTM5LWJjNTgtNGVmOS1iMTc4LWQ5MTRlODMzODA1ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00NDcyMTFkY2Y4MjYxYmNlMzVjNDhlYWRlMDQ4ZmI4ODMyZGRlZTY3ZjU5YmJmZTFlNDMxMWRlOWQyMjFhMDE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.LTQ4mo6iJ1cFfsJIs28q8E0SRscI3eYLbqEGT4V6OTs)
Workflow: simple.json
Samples:
![image](https://private-user-images.githubusercontent.com/133728487/269432912-0f8bfb87-83cb-4119-9777-e3948ec0cb5c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMyOTEyLTBmOGJmYjg3LTgzY2ItNDExOS05Nzc3LWUzOTQ4ZWMwY2I1Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05NGU5MTVlODcyMTZlM2ZkMzJkMWYzN2M3ZDQ4N2VjNjcyMmNlNjU2YzVlYzVlY2VmODI3ZjZkZjdkNTNjMmUzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.0xC_QCds0p641n6JcHcHUg7uOE3mr2NKQJwtVnIwFkE)
Workflow: sliding-window.json
Samples:
![]() |
![]() |
Upscale latent output using LatentUpscale
then do a 2nd pass with AnimateDiffSampler
.
![image](https://private-user-images.githubusercontent.com/133728487/268485612-987a1c5a-c1f8-4b24-8c62-f14496261d6c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3NjUxNDMsIm5iZiI6MTcxOTc2NDg0MywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDg1NjEyLTk4N2ExYzVhLWMxZjgtNGIyNC04YzYyLWYxNDQ5NjI2MWQ2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQxNjI3MjNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hZTQ4MWU2ZTMzMjdlMGJlNDU4Mjg2NjYyZjA2MDUyMDIyYzM5ZjQ5NzJiMDljZTJkMzM1ZTgxZjE3MzYwNTVlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.CGplZkrRAnj9uMB5ySbndShgfEqOt5c1qm1CGFi5FEg)
Workflow: latent-upscale.json
You will need following additional nodes:
- Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch
- Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors
- Use
LatentKeyframe
andTimestampKeyframe
from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index. - Use 2 controlnet modules for two images with weights reverted.
Workflow: cn-2images.json
Samples:
![]() |
![]() |
![]() |
![]() |
Using a GIF (or video, or a list of images) as ControlNet input.
Workflow: cn-vid2vid.json
Samples:
![]() |
![]() |
It's an xformers
bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with --disable-xformers
when booting ComfyUI.
Work around:
- Shorter your prompt and negative prompt
- Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
- Disable xformers with
--disable-xformers
See: continue-revolution/sd-webui-animatediff#31
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.