AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.
- Clone this repo into
custom_nodes
folder. - Download motion modules and put them under
comfyui-animatediff/models/
.
- Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk
- Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff
- AnimateDiff v2 mm_sd_v15_v2.ckpt
Download motion LoRAs and put them under comfyui-animatediff/loras/
folder.
Note: LoRAs only work with AnimateDiff v2 mm_sd_v15_v2.ckpt module.
![image](https://private-user-images.githubusercontent.com/133728487/270393962-7a9f62f7-702e-48a4-934c-bbfe1e23aff2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjcwMzkzOTYyLTdhOWY2MmY3LTcwMmUtNDhhNC05MzRjLWJiZmUxZTIzYWZmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xNTAzZTZlOGNhZDYyOTJjMGMxODVlZjM5NTQ3YzM4NzcxODg1NjViNjkwMWI0NWNjMzhiMzE3YzY2NjQ3Y2ZkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9._4necGjblnqFS-B45yV8l1mBdyBrpINIDSOyCkRNc9o)
Workflow: lora.json
Samples:
![]() |
![]() |
![]() |
![]() |
The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the SlidingWindowOptions
node. See the sample workflow bellow.
![image](https://private-user-images.githubusercontent.com/133728487/268482819-9d756d01-ea45-4d1c-8e48-56f2725c7ca1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODE5LTlkNzU2ZDAxLWVhNDUtNGQxYy04ZTQ4LTU2ZjI3MjVjN2NhMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jODM1YjFiYWM0NjdhYzAyMGM4OWFkODgyOTRkMDhlNTYyYWMzZTYwMGYzODI3ZGJiZmM4ZGU0ZGNjMjgyYTJmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.ZfMsd76wD4a_QPk-5iL_gfAbAI4N6O2rHv3-u4-_fBI)
- Mostly the same with
KSampler
motion_module
: useAnimateDiffLoader
to load the motion moduleinject_method
: should left defaultframe_number
: animation lengthlatent_image
: You can pass anEmptyLatentImage
sliding_window_opts
: custom sliding window options
![image](https://private-user-images.githubusercontent.com/133728487/269430269-a352195d-f40c-494d-bd3d-30ee88174b88.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMwMjY5LWEzNTIxOTVkLWY0MGMtNDk0ZC1iZDNkLTMwZWU4ODE3NGI4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02ZTJhZWVjOGJhMDZjNGUyNzIxYTc3ZmNiODViY2M1Mjg2MzIzMDM1M2ZkZjcyYjcwYWM2MjY3ZWZhYzc3NmMxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.g393e2f8mhrSxOGpiiYjapwDeQ01JEHNPQJP6qOuCaY)
- Combine GIF frames and produce the GIF image
frame_rate
: number of frame per secondloop_count
: use 0 for infinite loopsave_image
: should GIF be saved to diskformat
: supportsimage/gif
,image/webp
(better compression),video/webm
,video/h264-mp4
,video/h265-mp4
. To use video formats, you'll need ffmpeg installed and available inPATH
![image](https://private-user-images.githubusercontent.com/133728487/268482889-381c5acc-06ef-43da-ada0-3dc76f37a3e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODg5LTM4MWM1YWNjLTA2ZWYtNDNkYS1hZGEwLTNkYzc2ZjM3YTNlNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04MGU1NzVlNzQ3N2QxYjE5MzIwNmJjZDg5Zjk5YmM5ODg3OWU0NjQxMGE4NDlkMmE0MWQ0ZjRlZWY3NGIyZGRkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.SyYkjI8DWLLdsj3xf0YD-iS_2TZGZvYIT1xS3hZ3FS8)
Custom sliding window options
context_length
: number of frame per window. Use 16 to get the best results. Reduce it if you have low VRAM.context_stride
:- 1: sampling every frame
- 2: sampling every frame then every second frame
- 3: sampling every frame then every second frame then every third frames
- ...
context_overlap
: overlap frames between each window sliceclosed_loop
: make the GIF a closed loop, will add more sampling step
![image](https://private-user-images.githubusercontent.com/133728487/269431763-6679a8dd-bf96-419f-8934-ea2b046dd23c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMxNzYzLTY2NzlhOGRkLWJmOTYtNDE5Zi04OTM0LWVhMmIwNDZkZDIzYy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zYmM5MGU0OWM2MzU3ZDIxNTM4MmRhYzdkNzk4YzY3ZDE4M2Y4MjBkNDFmMWZhZjlhYTk4NzZmYjc0ZTg1NmNmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.fRhKjOWZ407Vy9TTVFUWqu-GtU1DK5-L1LFftlIp4oI)
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
frame_start
: Skip some begining frames and start atframe_start
frame_limit
: Only takeframe_limit
frames
![image](https://private-user-images.githubusercontent.com/133728487/269429598-684176d5-6369-4a27-9f33-e721e0fe1876.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDI5NTk4LTY4NDE3NmQ1LTYzNjktNGEyNy05ZjMzLWU3MjFlMGZlMTg3Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNDI3MmY0NDIwOWZiMzVjZmFiMmJjYmM0Y2FkZjA2ZWRiZTVkZTQ3YTAzZjJhYzM1NzU0NjY3YmU2YWM0ODgxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.rc9l1oNukPfYrs-a0URR0glDnntjsWeg_0pr45ns90w)
![image](https://private-user-images.githubusercontent.com/133728487/268311503-b7164539-bc58-4ef9-b178-d914e833805e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4MzExNTAzLWI3MTY0NTM5LWJjNTgtNGVmOS1iMTc4LWQ5MTRlODMzODA1ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0wZTc1Yjg3MmZkN2FjNDE1ZDkyMDk3NDNmNmNkOTNiZDZjMzIyNWFhZGI1YWNlYWZmZjY1OTg0MTAxNjY2NDM5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.Mpln9S20jjdc9j9Xyt6R2Wzv8266INeCIhB2l4sG-qs)
Workflow: simple.json
Samples:
![image](https://private-user-images.githubusercontent.com/133728487/269432912-0f8bfb87-83cb-4119-9777-e3948ec0cb5c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMyOTEyLTBmOGJmYjg3LTgzY2ItNDExOS05Nzc3LWUzOTQ4ZWMwY2I1Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNDMwNTczNzU4MmQyZWQ0ZDc2MWVlMDJlZGE5YjE1ZTc4NWVjYWE2NmQzNWM3ZmM2YjQwYTBmY2Q0YjBmZDFkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.CAaYIgR_7kWSWSH6UWHsEAFQr8q-edH7gxegj3bXuKE)
Workflow: sliding-window.json
Samples:
![]() |
![]() |
Upscale latent output using LatentUpscale
then do a 2nd pass with AnimateDiffSampler
.
![image](https://private-user-images.githubusercontent.com/133728487/268485612-987a1c5a-c1f8-4b24-8c62-f14496261d6c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MzE4OTgsIm5iZiI6MTcxOTYzMTU5OCwicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDg1NjEyLTk4N2ExYzVhLWMxZjgtNGIyNC04YzYyLWYxNDQ5NjI2MWQ2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjI5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYyOVQwMzI2MzhaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xMjQyZmY5OGY3Mzk0YjZhMTNkNjRiMzk0ZGZkN2Y3MDc2NjA4YzE5OWM1YWZhMTIwODg3OGI3ODdmYmUxNTg2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.JisJaSIQ7n5rQ5alv2_phGWsZwh3mrEymdAxdOL6pws)
Workflow: latent-upscale.json
You will need following additional nodes:
- Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch
- Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors
- Use
LatentKeyframe
andTimestampKeyframe
from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index. - Use 2 controlnet modules for two images with weights reverted.
Workflow: cn-2images.json
Samples:
![]() |
![]() |
![]() |
![]() |
Using a GIF (or video, or a list of images) as ControlNet input.
Workflow: cn-vid2vid.json
Samples:
![]() |
![]() |
It's an xformers
bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with --disable-xformers
when booting ComfyUI.
Work around:
- Shorter your prompt and negative prompt
- Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
- Disable xformers with
--disable-xformers
See: continue-revolution/sd-webui-animatediff#31
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.