[Test] new controlnet (animatediff_controlnet))
s9roll7 opened this issue · comments
sample_animatediff_controlnet2.mp4
original / animatediff_controlnet + controlnet_openpose(0.5) / animatediff_controlnet + controlnet_openpose(0.5)
1024x1280
animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize create-region PATH_TO_STYLIZE_DIR
animatediff stylize generate PATH_TO_STYLIZE_DIR
sample_animatediff_controlnet3.mp4
original /
animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter/
animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter
768x1152
animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize generate PATH_TO_STYLIZE_DIR
Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my
prompt.json. What could be wrong?
Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?
| Original | Generated |
|---|---|
input_.mp4 |
output_.mp4 |
Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my prompt.json. What could be wrong?
Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?
Original Generated
input_.mp4
output_.mp4
The json file looks fine.
I'm guessing 00_controlnet_image\animatediff_controlnet is empty?
If there's nothing in the directory, nothing will happen.
You need to copy all the images in controlnet_openpose to animatediff_controlnet.
sample_animatediff_controlnet4.mp4
original / animatediff_controlnet + controlnet_openpose
896x768
generation time 196sec (4090)
vram 10.3 GB
animatediff stylize create-config PATH_TO_MOVIE_FILE -gh
animatediff stylize generate PATH_TO_STYLIZE_DIR
use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM
use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM
"controlnet_map": {
"input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image",
"max_samples_on_vram": 200, # <-----------
"max_models_on_vram": 3, # <---------
"save_detectmap": true,
use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM
"controlnet_map": { "input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image", "max_samples_on_vram": 200, # <----------- "max_models_on_vram": 3, # <--------- "save_detectmap": true,
thank you！
Hello, I used the img2img function in Stable Diffusion 1.5, which only added prompt words. The effect is quite different from that of Animatediff cli prompt travel. How can I maintain better consistency with SD? Thank you
