The 3D avatar generation methods have been largely attracting attention as the market of metaverse applications grows significantly. Previous work has studied the 3D reconstruction methods that map the 2D human face images to the 3D objects, however, research on generating 3D cartoon-like avatars has gained little attention. In this paper, we present a deep learning-based pipeline that (1) changes the domain of original images to the cartoon domain that is selected by a user and (2) generates 3D avatar objects using the cartoon 2D image whose domain is changed. Moreover, as we apply the face alignment method to the CartoonStyleGAN, we have improved the quality of generated 2D cartoon images which results in improved 3D head reconstruction. We hope that our work can be a milestone for the research on generating 3D cartoon avatar objects based on 2D human face images.
Clone the repo:
https://github.com/noheat61/my_avatar_ai
cd my_avatar_ai- Python 3.9
- CUDA 11.3
- Linux(pytorch3d only support Linux)
# Install all packages(recommend using conda)
pip install -r requirements.txtYou can download all the files you need just by running the script frame below.
# Download all CartoonStyleGAN networks
python download_cartoon_data.py
# Download DECA template models
bash download_deca_model.shBefore run scripts below, input your own image in "images/".
# Make directories
mkdir cartoon_image avatar
# Inference
python example.py| Dataset | Path |
|---|---|
| FFHQ | https://github.com/rosinality/stylegan2-pytorch |
| StyleGAN2-Encoder | https://github.com/bryandlee/stylegan2-encoder-pytorch |
| Naver Webtoon | https://github.com/bryandlee/naver-webtoon-data |
| DISNEY | https://github.com/justinpinkney/toonify |
| COMICS | https://github.com/Sxela/face2comics |
| ART | https://github.com/NVlabs/metfaces-dataset |
We improved the accuracy of the projection by performing face-alignment preprocessing before putting the image as an input of CartoonStyleGAN.
For more details, please check our paper.
You can connect your own avatar head with 3D body we provide(modified from mixamo).
Due to the compatibility between packages, this function must be performed in a new environment from python 3.7.
# Install 3D packages(fbx, bpy)
pip install -r fbx_utils/requirements.txt
bpy_post_install
bash fbx_utils/install_fbx.sh
# Download 3D bodies
bash fbx_utils/download_body.sh
# Convert obj to fbx(3D heads)
python fbx_utils/obj2fbx.py
# Connect with body(default: 1.fbx)
python fbx_utils/example.py