sotaDetector / RiggingJs

Tensorflow.js and Three.js skeletal animation tool

Rigging.js

Rigging.js is an open-source react.js application, that takes the keypoints generated by the face mesh tensorflow.js model, then map the movement that the person is doing in front of the camera into a 3d model. Any model downloaded from https://www.mixamo.com/ can be used.

Build And Run

git clone https://github.com/haruiz/RiggingJs.git
cd RiggingJs
npm install
npm start

Try with different models

As mentioned above, any model downloaded from https://www.mixamo.com/ can be used. These models are freely available.

Roadmap

• Automatic Rigging and Animation of 3D Characters: using the current state of the art deep learning models.
• 3D expression animation using facemesh model
• Record animation

You can see the detailed roadmap here.

How to contribute:

• Feel free to send a pull request

Third party libraries:

• React.js
• Mathjs
• Three.js
• TensorFlow.js

Inspiration taken from:

• Pose Animator 2: Pose Animator takes a 2D vector illustration and animates its containing curves in real-time based on the recognition result from PoseNet and FaceMesh. It borrows the idea of skeleton-based animation from computer graphics and applies it to vector characters.
• PoseNet Model: This package contains a standalone model called PoseNet, as well as some demos, for running real-time pose estimation in the browser using TensorFlow.js.
• BodyPix: This package contains a standalone model called BodyPix, as well as some demos, for running real-time person and body part segmentation in the browser using TensorFlow.js.
• MediaPipe Facemesh: MediaPipe Facemesh is a lightweight machine learning pipeline predicting 486 3D facial landmarks to infer the approximate surface geometry of a human face (paper).