bochuanwu / OpenVtuber

虚拟爱抖露共享计划

OpenVtuber-虚拟爱抖露共享计划

Kizuna-Ai MMD demo : face capture via single RGB camera

Installation

Requirements

• Python 3.5+
• Linux, Windows or macOS
• mxnet (>=1.4)
• node.js and npm or yarn

While not required, for optimal performance(especially for the detector) it is highly recommended to run the code using a CUDA enabled GPU.

Run

• node ./NodeServer/server.js
• make -C ./PythonClient/rcnn/
• python3.7 ./PythonClient/vtuber_usb_camera.py --gpu -1

人脸检测 （Face Detection）

• RetinaFace: Single-stage Dense Face Localisation in the Wild
• RetinaFace (mxnet version)

RetinaFace is a practical single-stage SOTA face detector which is initially described in arXiv technical report

头部姿态估计（Head Pose Estimation）

• head-pose-estimation

特征点检测（Facial Landmarks Tracking）

The 2D pre-trained model is from the deep-face-alignment repository.

• Algorithm from TPAMI 2019
• Training set is based on i-bug 300-W datasets. It's annotation is shown below:
  
  

注视估计（Gaze Estimation)

• Laser Eye : Gaze Estimation via Deep Neural Networks

MMD Loader

• Three.js Webgl Loader

Live2D

• 插件版本
• 打包版本

Thanks

• threejs.org
• kizunaai.com

Citation

@article{Bulat2018Hierarchical,
  title={Hierarchical binary CNNs for landmark localization with limited resources},
  author={Bulat, Adrian and Tzimiropoulos, Yorgos},
  journal={IEEE Transactions on Pattern Analysis & Machine Intelligence},
  year={2018},
}
  
@inproceedings{deng2019retinaface,
title={RetinaFace: Single-stage Dense Face Localisation in the Wild},
author={Deng, Jiankang and Guo, Jia and Yuxiang, Zhou and Jinke Yu and Irene Kotsia and Zafeiriou, Stefanos},
booktitle={arxiv},
year={2019}
}