Carlyx / DPE

[CVPR 2023] DPE: Disentanglement of Pose and Expression for General Video Portrait Editing

DPE： Disentanglement of Pose and Expression for General Video Portrait Editing

           

Youxin Pang ^1,2,3   Yong Zhang ^3,*  Weize Quan ^1,2  Yanbo Fan ³  Xiaodong Cun ³  Ying Shan ³   Dong-ming Yan ^1,2,*  

¹ MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China   ² School of Artificial Intelligence, University of Chinese Academy of Sciences   ³ Tencent AI Lab, ShenZhen, China  

CVPR 2023

🔥 Demo

• 🔥 Video editing: single source video & a driving video & a piece of audio. We tranfer pose through the video and transfer expression through the audio with the help of SadTalker.

Source video	Result
full_s.mp4	dpe.mp4
full_s.mp4	dpe.mp4
full_s.mp4	dpe.mp4

• 🔥 Video editing: single source image & a driving video & a piece of audio. We tranfer pose through the video and transfer expression through the audio with the help of SadTalker.

demo4_1.mp4

demo5_1.mp4

• 🔥 Video editing: single source image & two driving videos. We tranfer pose through the first video and transfer expression through the second video. Some videos are selected from here.

📋 Changelog

• 2023.05.06 Support Enhancement.
• 2023.05.05 Support Video editing.
• 2023.04.30 Add some demos.
• 2023.03.18 Support Pose driving，Expression driving and Pose and Expression driving.
• 2023.03.18 Upload the pre-trained model, which is fine-tuning for expression generator.
• 2023.03.03 Release the test code!
• 2023.02.28 DPE has been accepted by CVPR 2023!

🚧 TODO

• Test code for video driving.
• Some demos.
• Gradio/Colab Demo.
• Training code of each componments.
• Test code for video editing.
• Integrate audio driven methods for video editing.
• Integrate GFPGAN for face enhancement.

🔮 Inference

Dependence Installation

CLICK ME

git clone https://github.com/Carlyx/DPE
cd DPE 
conda create -n dpe python=3.8
source activate dpe
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt
### install gpfgan for enhancer
pip install git+https://github.com/TencentARC/GFPGAN

Trained Models

CLICK ME

Please download our pre-trained model and put it in ./checkpoints.

Model	Description
checkpoints/dpe.pt	Pre-trained model (V1).

Expression driving

python run_demo.py --s_path ./data/s.mp4 \
 		--d_path ./data/d.mp4 \
		--model_path ./checkpoints/dpe.pt \
		--face exp \
		--output_folder ./res

Pose driving

python run_demo.py --s_path ./data/s.mp4 \
 		--d_path ./data/d.mp4 \
		--model_path ./checkpoints/dpe.pt \
		--face pose \
		--output_folder ./res

Expression and pose driving

python run_demo.py --s_path ./data/s.mp4 \
 		--d_path ./data/d.mp4 \
		--model_path ./checkpoints/dpe.pt \
		--face both \
		--output_folder ./res

Crop full video

python crop_video.py

Video editing

Before video editing, you should run python crop_video.py to process the input full video. For pre-trained segmentation model, you can download from here and put it in ./checkpoints.

(Optional) You can run git clone https://github.com/TencentARC/GFPGAN and download the pre-trained enhancement model from here and put it in ./checkpoints. Then you can use --EN to make the result better.

python run_demo_paste.py --s_path <cropped source video> \
  --d_path <driving video> \
  --box_path <txt after running crop_video.py> \
  --model_path ./checkpoints/dpe.pt \
  --face exp \
  --output_folder ./res \
  --EN 

Video editing for audio driving

  TODO

🛎 Citation

If you find our work useful in your research, please consider citing:

@article{pang2023dpe,
  title={DPE: Disentanglement of Pose and Expression for General Video Portrait Editing},
  author={Pang, Youxin and Zhang, Yong and Quan, Weize and Fan, Yanbo and Cun, Xiaodong and Shan, Ying and Yan, Dong-ming},
  journal={arXiv preprint arXiv:2301.06281},
  year={2023}
}

💗 Acknowledgements

Part of the code is adapted from LIA, PIRenderer, STIT. We thank authors for their contribution to the community.

🥂 Related Works

• StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN (ECCV 2022)
• CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior (CVPR 2023)
• VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild (SIGGRAPH Asia 2022)
• SadTalker： Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (CVPR 2023)
• 3D GAN Inversion with Facial Symmetry Prior (CVPR 2023)
• T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations (CVPR 2023)

📢 Disclaimer

This is not an official product of Tencent. This repository can only be used for personal/research/non-commercial purposes.