- Mar. 24, 2024. Support interactive demo with gradio.
- Mar. 13, 2024. Release the inference code.
- Mar. 12, 2024. Rep initialization.
We introduce DragAnything, which utilizes an entity representation to achieve motion control for any object in controllable video generation. Comparison to existing motion control methods, DragAnything offers several advantages. Firstly, trajectory-based is more user-friendly for interaction, when acquiring other guidance signals (\eg{} masks, depth maps) is labor-intensive. Users only need to draw a line~(trajectory) during interaction. Secondly, our entity representation serves as an open-domain embedding capable of representing any object, enabling the control of motion for diverse entities, including background. Lastly, our entity representation allows simultaneous and distinct motion control for multiple objects. Extensive experiments demonstrate that our DragAnything achieves state-of-the-art performance for FVD, FID, and User Study, particularly in terms of object motion control, where our method surpasses the previous state of the art (DragNUWA) by 26% in human voting.
Input Image | Drag point with SAM | 2D Gaussian Trajectory | Generated Video |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Model | Input Image and Drag | Generated Video | Visualization for Pixel Motion |
DragNUWA | ![]() |
![]() |
![]() |
Ours | ![]() |
![]() |
![]() |
DragNUWA | ![]() |
![]() |
![]() |
Ours | ![]() |
![]() |
![]() |
DragNUWA | ![]() |
![]() |
![]() |
Ours | ![]() |
![]() |
![]() |
Drag point with SAM | 2D Gaussian | Generated Video | Visualization for Pixel Motion |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
- Python >= 3.10 (Recommend to use Anaconda or Miniconda)
- PyTorch >= 1.13.0+cu11.7
git clone https://github.com/Showlab/DragAnything.git
cd DragAnything
conda create -n DragAnything python=3.8
conda activate DragAnything
pip install -r requirements.txt
Download VIPSeg and Youtube-VOS to the ./data
directory.
You can use our preprocessed annotation files or choose to process your own motion trajectory annotation files using Co-Track.
If you choose to generate motion trajectory annotations yourself, you need to follow the processing steps outlined in Co-Track.
cd ./utils/co-tracker
pip install -e .
pip install matplotlib flow_vis tqdm tensorboard
mkdir -p checkpoints
cd checkpoints
wget https://huggingface.co/facebook/cotracker/resolve/main/cotracker2.pth
cd ..
Then, modify the corresponding video_path
, ann_path
, and save_path
in the Generate_Trajectory_for_VIPSeg.sh
file, and run the command. The corresponding trajectory annotations will be saved as .json files in the save_path directory.
Generate_Trajectory_for_VIPSeg.sh
You can run the following command for visualization.
cd .utils/
python vis_trajectory.py
We adopt the ChilloutMix as pretrained model for extraction of entity representation, please download the diffusers version:
mkdir -p utils/pretrained_models
cd utils/pretrained_models
# Diffusers-version ChilloutMix to utils/pretrained_models
git-lfs clone https://huggingface.co/windwhinny/chilloutmix.git
And you can download our pretrained model for the controlnet:
mkdir -p model_out/DragAnything
cd model_out/DragAnything
# Diffusers-version DragAnything to model_out/DragAnything
git-lfs clone https://huggingface.co/weijiawu/DragAnything
cd .utils/
python extract_semantic_point.py
For VIPSeg
sh ./script/train_VIPSeg.sh
For YouTube VOS
sh ./script/train_youtube_vos.sh
Evaluation for FID
cd utils
sh Evaluation_FID.sh
Evaluation for Fréchet Video Distance (FVD)
cd utils/Eval_FVD
sh compute_fvd.sh
cd utils/Eval_ObjMC
python ./ObjMC.py
python demo.py
or run the interactive inference with gradio (install the gradio==3.50.2
).
cd ./script
download the weight of sam_vit_h_4b8939.pth
from SAM
python gradio_run.py
cd utils/co-tracker
python demo.py
@misc{wu2024draganything,
title={DragAnything: Motion Control for Anything using Entity Representation},
author={Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, Di Zhang},
year={2024},
eprint={2403.07420},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
- Thanks to Diffusers for the wonderful work and codebase.
- Thanks to svd-temporal-controlnet for the wonderful work and codebase.
- Thanks to chaojie for building ComfyUI-DragAnything.