Collaborative Neural Rendering using Anime Character Sheets

Demo Page | Colab English/中文 | Hugging Face | arXiv | BiliBili | YouTube

Introduction

This project is the official implement of Collaborative Neural Rendering using Anime Character Sheets, which aims to genarate vivid dancing videos from hand-drawn anime character sheets(ACS). Watch more demos and details in our firmly recommended video in BiliBili or YouTube.

Usage

Prerequisites

NVIDIA GPU + CUDA + CUDNN
Python 3.6

Installation

Clone this repository

git clone https://github.com/megvii-research/CoNR

Dependencies

To install all the dependencies, please run the following commands.

cd CoNR
pip install -r requirements.txt

Download Weights Download weights from Google Drive. Alternatively, you can download from Baidu Netdisk (password:RDxc).

mkdir weights && cd weights
gdown https://drive.google.com/uc?id=1M1LEpx70tJ72AIV2TQKr6NE_7mJ7tLYx
gdown https://drive.google.com/uc?id=1YvZy3NHkJ6gC3pq_j8agcbEJymHCwJy0
gdown https://drive.google.com/uc?id=1AOWZxBvTo9nUf2_9Y7Xe27ZFQuPrnx9i
gdown https://drive.google.com/uc?id=19jM1-GcqgGoE1bjmQycQw_vqD9C5e-Jm

Prepare Inputs

We provide two Ultra-Dense Pose sequences for two characters. You can generate more UDPs via 3D models and motions refers to our paper, or use MMD2UDP(Thanks to @KurisuMakise004 ). Baidu Netdisk (password:RDxc)

# for short hair girl
gdown https://drive.google.com/uc?id=11HMSaEkN__QiAZSnCuaM6GI143xo62KO
unzip short_hair.zip
mv short_hair/ poses/

# for double ponytail girl
gdown https://drive.google.com/uc?id=1WNnGVuU0ZLyEn04HzRKzITXqib1wwM4Q
unzip double_ponytail.zip
mv double_ponytail/ poses/

We provide sample inputs of anime character sheets. You can also draw more by yourself. Character sheets need to be cut out from the background and in png format. Baidu Netdisk (password:RDxc)

# for short hair girl
gdown https://drive.google.com/uc?id=1r-3hUlENSWj81ve2IUPkRKNB81o9WrwT
unzip short_hair_images.zip
mv short_hair_images/ character_sheet/

# for double ponytail girl
gdown https://drive.google.com/uc?id=1XMrJf9Lk_dWgXyTJhbEK2LZIXL9G3MWc
unzip double_ponytail_images.zip
mv double_ponytail_images/ character_sheet/

RUN!

with web UI (powered by Streamlit)

streamlit run streamlit.py --server.port=8501

then open your browser and visit localhost:8501, follow the instructions to generate video.

via terminal

mkdir {dir_to_save_result}

python -m torch.distributed.launch \
--nproc_per_node=1 train.py --mode=test \
--world_size=1 --dataloaders=2 \
--test_input_poses_images={dir_to_poses} \
--test_input_person_images={dir_to_character_sheet} \
--test_output_dir={dir_to_save_result} \
--test_checkpoint_dir={dir_to_weights}

ffmpeg -r 30 -y -i {dir_to_save_result}/%d.png -r 30 -c:v libx264 output.mp4 -r 30

Citation

@article{lin2022conr,
  title={Collaborative Neural Rendering using Anime Character Sheets},
  author={Lin, Zuzeng and Huang, Ailin and Huang, Zhewei and Hu, Chen and Zhou, Shuchang},
  journal={arXiv preprint arXiv:2207.05378},
  year={2022}
}

BeyondYourself / CoNR